Nov 02 12:26:22.039301 ip-10-0-134-229 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 12:26:22.469132 ip-10-0-134-229 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:22.469132 ip-10-0-134-229 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 12:26:22.469132 ip-10-0-134-229 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:22.469132 ip-10-0-134-229 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:26:22.473206 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473125    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:22.473206 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473205    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:22.473206 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473209    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473215    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473219    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473223    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473226    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473229    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473232    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473235    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473237    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473240    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473242    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473245    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473247    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473249    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473252    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473254    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473257    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473259    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473262    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:22.473297 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473264    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473266    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473269    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473277    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473280    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473283    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473285    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473288    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473290    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473293    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473295    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473298    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473301    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473303    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473306    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473309    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473313    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473315    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473317    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473320    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:22.473730 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473322    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473325    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473327    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473330    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473333    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473335    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473337    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473340    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473342    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473344    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473347    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473349    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473352    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473354    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473357    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473362    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473365    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473369    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473371    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:22.474261 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473375    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473378    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473380    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473383    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473385    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473388    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473390    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473394    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473396    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473399    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473402    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473404    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473407    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473410    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473413    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473415    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473418    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473420    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473423    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473425    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:22.474707 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473428    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473430    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473433    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473435    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473437    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473440    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473442    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473446    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473449    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473451    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473455    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473457    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473461    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473464    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473466    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473468    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473471    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473473    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473476    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473478    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:22.475183 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473481    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473483    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473486    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473488    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473491    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.473493    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474272    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474279    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474282    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474285    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474288    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474290    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474293    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474298    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474302    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474305    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474308    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474311    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474314    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:22.475653 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474317    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474320    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474323    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474326    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474328    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474331    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474333    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474336    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474339    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474341    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474344    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474346    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474349    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474351    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474354    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474357    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474359    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474362    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474364    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:22.476225 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474368    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474370    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474373    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474375    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474378    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474380    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474383    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474385    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474388    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474390    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474393    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474395    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474398    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474401    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474404    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474406    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474409    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474412    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474415    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:22.476672 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474418    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474420    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474423    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474426    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474428    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474431    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474433    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474436    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474438    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474440    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474443    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474445    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474448    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474450    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474453    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474456    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474458    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474461    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474463    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474466    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:22.477135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474468    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474471    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474473    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474476    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474478    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474481    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474485    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474487    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474490    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474493    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474495    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474498    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474501    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474503    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474505    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474508    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474510    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474513    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474515    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474518    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:22.477607 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474520    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474523    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474525    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474528    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474530    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474534    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474538    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474541    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474545    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474547    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474550    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474553    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474555    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474558    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.474561    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475056    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475061    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475064    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475067    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:22.478126 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475071    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475073    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475076    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475078    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475081    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475084    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475086    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475089    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475091    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475094    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475096    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475099    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475102    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475104    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475106    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475109    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475112    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475115    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475117    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475119    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:22.478633 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475123    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475126    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475129    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475131    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475134    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475136    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475139    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475142    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475145    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475148    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475151    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475154    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475157    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475160    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475162    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475165    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475167    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475170    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475172    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475175    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:22.479200 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475177    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475180    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475182    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475185    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475188    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475191    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475194    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475196    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475199    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475201    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475204    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475207    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475210    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475214    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475217    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475219    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475222    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475224    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475227    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475229    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:22.479738 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475232    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475234    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475237    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475240    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475243    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475246    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475248    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475251    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475254    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475256    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475258    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475261    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475263    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475266    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475270    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475273    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475276    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475278    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475280    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475283    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:22.480301 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475286    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475288    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475291    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475293    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475296    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475298    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475302    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475305    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475307    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475310    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475312    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475315    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475317    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475319    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475324    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475328    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475331    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475333    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475336    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:22.480848 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475338    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475341    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.475343    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476281    2582 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476291    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476299    2582 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476304    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476310    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476313    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476317    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476321    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476324    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476327    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476331    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476334    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476337    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476341    2582 flags.go:64] FLAG: --cgroup-root=""
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476344    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476347    2582 flags.go:64] FLAG: --client-ca-file=""
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476350    2582 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476354    2582 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476358    2582 flags.go:64] FLAG: --cluster-domain=""
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476361    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 12:26:22.481373 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476364    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476367    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476370    2582 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476374    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476377    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476380    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476384    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476386    2582 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476389    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476393    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476396    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476399    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476403    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476406    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476409    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476412    2582 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476415    2582 flags.go:64] FLAG: --enable-server="true"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476418    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476423    2582 flags.go:64] FLAG: --event-burst="100"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476426    2582 flags.go:64] FLAG: --event-qps="50"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476429    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476433    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476436    2582 flags.go:64] FLAG: --eviction-hard=""
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476440    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 12:26:22.482005 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476443    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476446    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476449    2582 flags.go:64] FLAG: --eviction-soft=""
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476452    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476455    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476458    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476462    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476465    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476468    2582 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476471    2582 flags.go:64] FLAG: --feature-gates=""
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476476    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476479    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476482    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476485    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476488    2582 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476491    2582 flags.go:64] FLAG: --help="false"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476495    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476498    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476501    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476504    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476507    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476511    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476514    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 12:26:22.482656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476517    2582 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476520    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476523    2582 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476526    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476530    2582 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476532    2582 flags.go:64] FLAG: --kube-reserved=""
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476535    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476538    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476542    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476544    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476547    2582 flags.go:64] FLAG: --lock-file=""
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476550    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476553    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476556    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476562    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476565    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476572    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476575    2582 flags.go:64] FLAG: --logging-format="text"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476578    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476582    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476585    2582 flags.go:64] FLAG: --manifest-url=""
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476588    2582 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476597    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476600    2582 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476604    2582 flags.go:64] FLAG: --max-pods="110"
Nov 02 12:26:22.483248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476607    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476610    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476613    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476616    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476619    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476622    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476625    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476632    2582 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476635    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476638    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476641    2582 flags.go:64] FLAG: --pod-cidr=""
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476644    2582 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476647    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476650    2582 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476653    2582 flags.go:64] FLAG: --port="10250"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476656    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476659    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0486a016b8c5edbbe"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476662    2582 flags.go:64] FLAG: --qos-reserved=""
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476665    2582 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476668    2582 flags.go:64] FLAG: --register-node="true"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476671    2582 flags.go:64] FLAG: --register-with-taints=""
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476675    2582 flags.go:64] FLAG: --registry-burst="10"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476678    2582 flags.go:64] FLAG: --registry-qps="5"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476682    2582 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476688    2582 flags.go:64] FLAG: --reserved-memory=""
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476694    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 12:26:22.483858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476698    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476701    2582 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476704    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476707    2582 flags.go:64] FLAG: --runonce="false"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476709    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476712    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476715    2582 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476718    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476721    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476724    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476727    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476730    2582 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476733    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476736    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476742    2582 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476746    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476749    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476752    2582 flags.go:64] FLAG: --system-cgroups=""
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476755    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476761    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476764    2582 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476766    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476770    2582 flags.go:64] FLAG: --tls-min-version=""
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476773    2582 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476776    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 12:26:22.484516 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476779    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476784    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476787    2582 flags.go:64] FLAG: --v="2"
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476791    2582 flags.go:64] FLAG: --version="false"
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476795    2582 flags.go:64] FLAG: --vmodule=""
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476799    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.476803    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476940    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476943    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476946    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476949    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476952    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476954    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476957    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476959    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476962    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476964    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476967    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476969    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476972    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476974    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:22.485138 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476978    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476981    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476984    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.476987    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477002    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477005    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477007    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477010    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477013    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477015    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477018    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477020    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477024    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477027    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477029    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477032    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477035    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477037    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477041    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477043    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:22.485618 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477046    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477049    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477053    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477057    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477060    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477063    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477065    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477068    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477071    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477074    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477077    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477081    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477085    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477088    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477091    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477094    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477097    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477100    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477103    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:22.486100 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477106    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477109    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477112    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477115    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477117    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477121    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477123    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477126    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477128    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477131    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477134    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477136    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477140    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477143    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477145    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477148    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477151    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477153    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477156    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477158    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:22.486545 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477160    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477163    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477165    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477168    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477170    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477174    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477176    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477179    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477182    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477185    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477187    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477190    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477192    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477195    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477197    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477200    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477202    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477206    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477209    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477211    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:22.487084 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477214    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477216    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477219    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477221    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477224    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477227    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477229    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477232    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477234    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477236    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477239    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477242    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477244    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.477271    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477306    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477311    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:22.487843 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477314    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477317    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477321    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477323    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477326    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477329    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477331    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477334    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477337    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477339    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477342    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477344    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477347    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477351    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477353    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477356    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477358    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477361    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477364    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:22.488325 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477366    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477369    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477372    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477375    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477377    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477380    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477383    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477385    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477388    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477390    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477392    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477395    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477397    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477400    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477403    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477406    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477409    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477411    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477414    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477417    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:22.488794 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477419    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477422    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477425    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477427    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477430    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477433    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477437    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477439    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477442    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477444    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477447    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477449    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477452    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477454    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477457    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477460    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477463    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477465    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477468    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477470    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:22.489291 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477473    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477475    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477478    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477480    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477483    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477485    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477488    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477492    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477495    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477499    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477503    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477506    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477509    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477513    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477516    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477518    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477521    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477524    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477529    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:22.489747 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477532    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477535    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477537    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477540    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477543    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477545    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477549    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477551    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477554    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477556    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477559    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477562    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477564    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477567    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477569    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477571    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477574    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477576    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477579    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477582    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:22.490229 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477585    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477588    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477591    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477593    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477595    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.477598    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.484880    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.484898    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.484911    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.484916    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.484972    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.484978    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.484981    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.484984    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.484987    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485007    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485011    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485013    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485017    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:22.490693 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485020    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485024    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485029    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485033    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485036    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485039    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485042    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485045    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485048    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485050    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485053    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485055    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485058    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485060    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485063    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485065    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485068    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485071    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485073    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:22.491276 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485076    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485080    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485083    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485085    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485088    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485091    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485093    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485096    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485098    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485101    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485104    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485106    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485108    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485111    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485113    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485116    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485118    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485121    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485123    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485126    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:22.491712 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485128    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485131    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485133    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485137    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485140    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485143    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485146    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485149    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485151    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485154    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485157    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485159    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485162    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485165    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485169    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485172    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485174    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485177    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485179    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:22.492198 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485182    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485185    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485187    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485190    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485192    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485194    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485197    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485199    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485202    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485204    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485206    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485209    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485211    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485214    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485217    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485219    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485222    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485224    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485227    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485229    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:22.492642 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485241    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485244    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485246    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485248    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485251    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485254    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485257    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485259    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485262    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485265    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485267    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485270    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485273    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485275    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485278    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485280    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485283    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485285    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:22.493135 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485288    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.485317    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485452    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485457    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485460    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485463    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485465    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485468    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485470    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485472    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485475    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485477    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485479    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485482    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485484    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485487    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:22.493549 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485489    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485492    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485494    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485497    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485501    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485503    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485506    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485509    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485511    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485514    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485517    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485521    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485523    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485526    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485529    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485532    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485534    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485537    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485539    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:22.493907 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485542    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485544    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485546    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485549    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485551    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485553    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485556    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485558    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485561    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485563    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485566    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485568    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485570    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485573    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485575    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485578    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485580    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485583    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485586    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485588    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:22.494355 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485591    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485594    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485596    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485599    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485601    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485604    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485606    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485608    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485611    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485613    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485616    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485618    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485621    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485623    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485626    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485628    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485631    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485634    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485636    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485639    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:22.494817 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485641    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485643    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485646    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485648    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485651    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485653    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485656    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485658    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485661    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485665    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485668    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485671    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485674    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485676    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485679    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485682    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485684    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485687    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485689    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:22.495416 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485692    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485694    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485696    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485699    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485701    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485704    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485707    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485709    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485711    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485714    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485716    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485719    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485722    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485724    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.485747    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485775    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:26:22.495859 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485778    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485781    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485785    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485788    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485791    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485795    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485799    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485802    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485805    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485808    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485810    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485813    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485816    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485818    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485821    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485823    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485826    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485828    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485831    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:26:22.496256 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485833    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485836    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485838    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485841    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485843    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485845    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485848    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485850    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485853    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485855    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485858    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485860    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485862    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485864    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485867    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485870    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485872    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485874    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485877    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485879    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:26:22.496689 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485882    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485884    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485887    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485889    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485892    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485894    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485897    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485899    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485902    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485904    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485907    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485909    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485911    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485914    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485918    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485921    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485924    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485926    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485929    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:26:22.497207 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485931    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485933    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485936    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485938    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485941    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485943    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485945    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485949    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485951    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485954    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485956    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485959    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485961    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485964    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485967    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485969    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485972    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485974    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485976    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485979    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:26:22.497644 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485982    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485984    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.485987    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486009    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486012    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486015    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486018    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486020    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486023    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486025    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486028    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486030    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486033    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486035    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486038    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486040    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486043    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486045    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486048    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:26:22.498163 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486050    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486053    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486055    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486058    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486060    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486064    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486067    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:22.486069    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.486126    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.490353    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.491438    2582 server.go:1014] "Starting client certificate rotation"
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.491527    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.491571    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.493292    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:26:22.498590 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.496008    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:26:22.505297 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.505280    2582 log.go:25] "Validated CRI v1 runtime API"
Nov 02 12:26:22.508323 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.508306    2582 log.go:25] "Validated CRI v1 image API"
Nov 02 12:26:22.510311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.510284    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 12:26:22.514017 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.513978    2582 fs.go:135] Filesystem UUIDs: map[64a92448-e7b0-477f-8772-79b4d7158201:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 feb98b30-4d11-401b-be6b-4da7db1e3340:/dev/nvme0n1p3]
Nov 02 12:26:22.514112 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.514017    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 12:26:22.518346 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.518224    2582 manager.go:217] Machine: {Timestamp:2026-11-02 12:26:22.517711296 +0000 UTC m=+0.354093166 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100131 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25f5866a5f816750b37ff752a560c7 SystemUUID:ec25f586-6a5f-8167-50b3-7ff752a560c7 BootID:eebfa388-deb3-4929-9203-04310d8b1020 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fa:81:df:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fa:81:df:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:d7:cf:64:08:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 12:26:22.518346 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.518332    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 12:26:22.518527 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.518446    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 12:26:22.521647 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.521620    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 12:26:22.521790 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.521649    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-229.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 12:26:22.521836 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.521793    2582 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 12:26:22.521836 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.521801    2582 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 12:26:22.521836 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.521826    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 12:26:22.521912 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.521848    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 12:26:22.522703 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.522693    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 12:26:22.525783 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.525770    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 12:26:22.525912 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.525903    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 12:26:22.526816 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.526805    2582 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 12:26:22.526858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.526828    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 12:26:22.526858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.526843    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 12:26:22.526923 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.526913    2582 kubelet.go:405] "Adding apiserver pod source"
Nov 02 12:26:22.526957 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.526925    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 12:26:22.527569 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.527550    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 12:26:22.530842 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.530821    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 12:26:22.535970 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.535954    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 12:26:22.536729 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.536715    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 12:26:22.536787 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.536746    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 12:26:22.536922 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.536911    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 12:26:22.536970 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.536927    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 12:26:22.536970 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.536934    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 12:26:22.536970 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.536939    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 12:26:22.536970 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.536945    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 12:26:22.536970 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.536950    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 12:26:22.537464 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.537450    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 12:26:22.537464 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.537462    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 12:26:22.537549 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.537473    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 12:26:22.537549 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.537483    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 12:26:22.537549 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.537492    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 12:26:22.537549 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.537502    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 12:26:22.538917 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.538907    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 12:26:22.538917 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.538917    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 12:26:22.542865 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.542851    2582 server.go:1263] "Started kubelet"
Nov 02 12:26:22.543028 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.542938    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 12:26:22.543078 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.543070    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 12:26:22.543127 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.543101    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 12:26:22.543843 ip-10-0-134-229 systemd[1]: Started Kubernetes Kubelet.
Nov 02 12:26:22.544477 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.544192    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 12:26:22.545710 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.545698    2582 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 12:26:22.548048 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.548022    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 12:26:22.550104 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.550089    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 12:26:22.551268 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.551240    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 12:26:22.551450 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.551429    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-229.ec2.internal\" not found"
Nov 02 12:26:22.551887 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.551871    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 12:26:22.551887 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.551880    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 12:26:22.552033 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.551894    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 12:26:22.552033 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.552011    2582 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 12:26:22.552033 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.552021    2582 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 12:26:22.552298 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.552284    2582 factory.go:153] Registering CRI-O factory
Nov 02 12:26:22.552345 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.552304    2582 factory.go:223] Registration of the crio container factory successfully
Nov 02 12:26:22.552380 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.552352    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 12:26:22.552380 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.552362    2582 factory.go:55] Registering systemd factory
Nov 02 12:26:22.552380 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.552371    2582 factory.go:223] Registration of the systemd container factory successfully
Nov 02 12:26:22.552463 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.552388    2582 factory.go:103] Registering Raw factory
Nov 02 12:26:22.552463 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.552398    2582 manager.go:1199] Started watching for new ooms in manager
Nov 02 12:26:22.553308 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.553256    2582 manager.go:319] Starting recovery of all containers
Nov 02 12:26:22.565544 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.565521    2582 manager.go:324] Recovery completed
Nov 02 12:26:22.569626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.569613    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:22.571666 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.571643    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:22.571757 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.571681    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:22.571757 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.571692    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:22.572287 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.572273    2582 cpu_manager.go:226] "Starting" policy="none"
Nov 02 12:26:22.572287 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.572284    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 12:26:22.572363 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.572285    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:22.572363 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.572301    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 12:26:22.572938 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.572925    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:22.572986 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.572951    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:22.572986 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.572960    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:22.573427 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.573415    2582 policy_none.go:50] "Start"
Nov 02 12:26:22.573465 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.573431    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 12:26:22.573465 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.573442    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 12:26:22.574575 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.574565    2582 policy_none.go:44] "Start"
Nov 02 12:26:22.577203 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.577022    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 12:26:22.577203 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.577019    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-229.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 12:26:22.577950 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.577029    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-229.ec2.internal.18e43a0dcca79f37  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-229.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-229.ec2.internal,},FirstTimestamp:2026-11-02 12:26:22.542823223 +0000 UTC m=+0.379205094,LastTimestamp:2026-11-02 12:26:22.542823223 +0000 UTC m=+0.379205094,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-229.ec2.internal,}"
Nov 02 12:26:22.578483 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.578461    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 12:26:22.578565 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.578513    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h854p"
Nov 02 12:26:22.587031 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.587014    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h854p"
Nov 02 12:26:22.594398 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.594372    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 12:26:22.594694 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.594602    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 12:26:22.594694 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.594667    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-229.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 12:26:22.594815 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.594628    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-229.ec2.internal.18e43a0dce5fbe59  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-229.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-229.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-229.ec2.internal,},FirstTimestamp:2026-11-02 12:26:22.571667033 +0000 UTC m=+0.408048903,LastTimestamp:2026-11-02 12:26:22.571667033 +0000 UTC m=+0.408048903,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-229.ec2.internal,}"
Nov 02 12:26:22.614642 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.614592    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 12:26:22.619144 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.619127    2582 manager.go:341] "Starting Device Plugin manager"
Nov 02 12:26:22.619249 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.619158    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 12:26:22.619249 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.619167    2582 server.go:87] "Starting device plugin registration server"
Nov 02 12:26:22.619430 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.619418    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 12:26:22.619477 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.619431    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 12:26:22.619542 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.619528    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 12:26:22.619605 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.619594    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 12:26:22.619646 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.619605    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 12:26:22.620234 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.620212    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 12:26:22.620322 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.620268    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-229.ec2.internal\" not found"
Nov 02 12:26:22.694902 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.694830    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 12:26:22.696587 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.696197    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 12:26:22.696587 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.696217    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 12:26:22.696587 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.696237    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 12:26:22.696587 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.696270    2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 12:26:22.709195 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.709172    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:22.719915 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.719898    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:22.720963 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.720949    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:22.721054 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.720976    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:22.721054 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.720986    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:22.721054 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.721024    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.738600 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.738580    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.738682 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.738605    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-229.ec2.internal\": node \"ip-10-0-134-229.ec2.internal\" not found"
Nov 02 12:26:22.778623 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.778592    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-229.ec2.internal\" not found"
Nov 02 12:26:22.796406 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.796370    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"]
Nov 02 12:26:22.796541 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.796448    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:22.798027 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.798010    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:22.798117 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.798045    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:22.798117 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.798060    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:22.799972 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.799957    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:22.800133 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.800119    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.800178 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.800149    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:22.800761 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.800744    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:22.800830 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.800776    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:22.800830 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.800788    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:22.800830 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.800812    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:22.800966 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.800834    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:22.800966 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.800847    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:22.802672 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.802657    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.802753 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.802685    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:26:22.803378 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.803364    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:26:22.803466 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.803392    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:26:22.803466 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.803406    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:26:22.839900 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.839882    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-229.ec2.internal\" not found" node="ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.842642 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.842627    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-229.ec2.internal\" not found" node="ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.852796 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.852777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55b1ac54009e3b5ad6df9f2e917c2d8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" (UID: \"55b1ac54009e3b5ad6df9f2e917c2d8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.852856 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.852810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55b1ac54009e3b5ad6df9f2e917c2d8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" (UID: \"55b1ac54009e3b5ad6df9f2e917c2d8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.852905 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.852833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2aebf6f2caaa3c2095ff6e33664045c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-229.ec2.internal\" (UID: \"f2aebf6f2caaa3c2095ff6e33664045c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.878910 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.878884    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-229.ec2.internal\" not found"
Nov 02 12:26:22.953079 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.953010    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55b1ac54009e3b5ad6df9f2e917c2d8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" (UID: \"55b1ac54009e3b5ad6df9f2e917c2d8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.953079 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.953028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55b1ac54009e3b5ad6df9f2e917c2d8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" (UID: \"55b1ac54009e3b5ad6df9f2e917c2d8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.953230 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.953093    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55b1ac54009e3b5ad6df9f2e917c2d8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" (UID: \"55b1ac54009e3b5ad6df9f2e917c2d8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.953230 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.953119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2aebf6f2caaa3c2095ff6e33664045c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-229.ec2.internal\" (UID: \"f2aebf6f2caaa3c2095ff6e33664045c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.953230 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.953169    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55b1ac54009e3b5ad6df9f2e917c2d8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" (UID: \"55b1ac54009e3b5ad6df9f2e917c2d8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.953230 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:22.953186    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2aebf6f2caaa3c2095ff6e33664045c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-229.ec2.internal\" (UID: \"f2aebf6f2caaa3c2095ff6e33664045c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:22.979139 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:22.979112    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-229.ec2.internal\" not found"
Nov 02 12:26:23.079371 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:23.079335    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-229.ec2.internal\" not found"
Nov 02 12:26:23.143578 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.143549    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:23.145692 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.145676    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:23.180244 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:23.180208    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-229.ec2.internal\" not found"
Nov 02 12:26:23.280847 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:23.280753    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-229.ec2.internal\" not found"
Nov 02 12:26:23.381295 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:23.381254    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-229.ec2.internal\" not found"
Nov 02 12:26:23.482065 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:23.482033    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-229.ec2.internal\" not found"
Nov 02 12:26:23.491236 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.491208    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 12:26:23.491410 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.491368    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 12:26:23.549827 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.549764    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:23.550785 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.550768    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 12:26:23.552402 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.552375    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:23.580370 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.580346    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:23.580821 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.580802    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:23.582530 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.582516    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:26:23.591191 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.591158    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 12:21:23 +0000 UTC" deadline="2028-06-09 17:46:02.501112402 +0000 UTC"
Nov 02 12:26:23.591191 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.591186    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14045h19m38.909930077s"
Nov 02 12:26:23.604231 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.604205    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:26:23.612215 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.612192    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 12:26:23.612337 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.612258    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qllrq"
Nov 02 12:26:23.629032 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:23.628976    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b1ac54009e3b5ad6df9f2e917c2d8b.slice/crio-b7b8bfaaab6e0d96223a2d994677ddcd08d7d624ffa75453543f3dc158adaf4e WatchSource:0}: Error finding container b7b8bfaaab6e0d96223a2d994677ddcd08d7d624ffa75453543f3dc158adaf4e: Status 404 returned error can't find the container with id b7b8bfaaab6e0d96223a2d994677ddcd08d7d624ffa75453543f3dc158adaf4e
Nov 02 12:26:23.636386 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.636370    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:26:23.639165 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:23.639137    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2aebf6f2caaa3c2095ff6e33664045c.slice/crio-7384673b07504ae5d5827772219deea4aa974c328d8347c76298f5bb4884961d WatchSource:0}: Error finding container 7384673b07504ae5d5827772219deea4aa974c328d8347c76298f5bb4884961d: Status 404 returned error can't find the container with id 7384673b07504ae5d5827772219deea4aa974c328d8347c76298f5bb4884961d
Nov 02 12:26:23.664580 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.664552    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:23.699906 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.699859    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal" event={"ID":"f2aebf6f2caaa3c2095ff6e33664045c","Type":"ContainerStarted","Data":"7384673b07504ae5d5827772219deea4aa974c328d8347c76298f5bb4884961d"}
Nov 02 12:26:23.700895 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:23.700872    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal" event={"ID":"55b1ac54009e3b5ad6df9f2e917c2d8b","Type":"ContainerStarted","Data":"b7b8bfaaab6e0d96223a2d994677ddcd08d7d624ffa75453543f3dc158adaf4e"}
Nov 02 12:26:24.349213 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.349123    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:26:24.528288 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.528241    2582 apiserver.go:52] "Watching apiserver"
Nov 02 12:26:24.564982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.564943    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 12:26:24.565391 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.565368    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-fspl8","openshift-multus/network-metrics-daemon-2vkgz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql","openshift-ovn-kubernetes/ovnkube-node-l9dwk","kube-system/global-pull-secret-syncer-x7jpb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal","openshift-network-diagnostics/network-check-target-2hzq7","openshift-network-operator/iptables-alerter-c4mfn","kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9w9rr","openshift-dns/node-resolver-k49jr","openshift-image-registry/node-ca-mdzbn","openshift-multus/multus-5qqx4","kube-system/konnectivity-agent-26p8v"]
Nov 02 12:26:24.568558 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.568185    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:24.568558 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:24.568285    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:24.570670 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.570642    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.572954 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.572929    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.575530 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.575506    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.577967 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.577940    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:24.578136 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:24.578039    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:24.580773 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.580419    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:26:24.583482 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.583463    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:26:24.586442 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.586288    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.590983 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.590962    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 12:26:24.591106 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.590963    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 12:26:24.591106 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.591060    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 12:26:24.591106 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.591060    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 12:26:24.591284 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.591098    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 12:26:24.591284 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.591185    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wm9fw\""
Nov 02 12:26:24.592086 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.592066    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:24.592203 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:24.592145    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:24.595451 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.595410    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.597731 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.597134    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 12:26:24.597731 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.597189    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 12:26:24.597731 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.597389    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 12:26:24.597731 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.597495    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2bc7v\""
Nov 02 12:26:24.597731 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.597569    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 12:26:24.597731 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.597597    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 12:26:24.597731 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.597626    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 12:26:24.597731 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.597648    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 12:26:24.597731 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.597670    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 12:26:24.597731 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.597699    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7btt2\""
Nov 02 12:26:24.597731 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.597731    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 12:26:24.598335 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.597810    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:26:24.600046 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.599992    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:26:24.600763 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.600743    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l5rh2\""
Nov 02 12:26:24.600842 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.600809    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 12:26:24.602505 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.602490    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 12:26:24.602599 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.602547    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:24.602697 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.602654    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hrc62\""
Nov 02 12:26:24.610475 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.610454    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:24.614817 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.614782    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 12:21:24 +0000 UTC" deadline="2028-08-14 07:23:36.874110246 +0000 UTC"
Nov 02 12:26:24.614817 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.614816    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15618h57m12.259298289s"
Nov 02 12:26:24.653258 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.653235    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 12:26:24.664445 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664410    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-systemd-units\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.664613 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:24.664613 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-tuned\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.664613 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664502    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-multus-certs\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.664613 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-socket-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.664613 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.664613 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwdjq\" (UniqueName: \"kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq\") pod \"network-check-target-2hzq7\" (UID: \"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60\") " pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:24.664613 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-node-log\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.664613 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysctl-conf\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d-konnectivity-ca\") pod \"konnectivity-agent-26p8v\" (UID: \"83dcdc7a-f1ba-4570-bed5-a40b0d3b213d\") " pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664667    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-systemd\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664689    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv5cq\" (UniqueName: \"kubernetes.io/projected/0162c6de-30e1-4e0b-949e-43b813b1e953-kube-api-access-pv5cq\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-daemon-config\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5x28\" (UniqueName: \"kubernetes.io/projected/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-kube-api-access-c5x28\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664788    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-env-overrides\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d59c905e-0042-4777-a2fe-d5d5ae04e47d-host\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664849    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664866    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-systemd\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-var-lib-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-lib-modules\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.664946 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgpqc\" (UniqueName: \"kubernetes.io/projected/d59c905e-0042-4777-a2fe-d5d5ae04e47d-kube-api-access-vgpqc\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-system-cni-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.664981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-cni-multus\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-kubelet\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-ovn\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665096    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysconfig\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665113    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-kubelet\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665131    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-sys-fs\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665171    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovnkube-script-lib\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665198    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97wkz\" (UniqueName: \"kubernetes.io/projected/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-kube-api-access-97wkz\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665227    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/259d31ef-0ef7-4655-9121-88fd07468c38-dbus\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665246    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-tmp\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-k8s-cni-cncf-io\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-netns\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665328    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-host-slash\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:26:24.665700 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-cni-bin\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665400    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d59c905e-0042-4777-a2fe-d5d5ae04e47d-serviceca\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-cnibin\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmb9\" (UniqueName: \"kubernetes.io/projected/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-kube-api-access-wvmb9\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-cni-netd\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-var-lib-kubelet\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovnkube-config\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665588    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovn-node-metrics-cert\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665613    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d-agent-certs\") pod \"konnectivity-agent-26p8v\" (UID: \"83dcdc7a-f1ba-4570-bed5-a40b0d3b213d\") " pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-modprobe-d\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-sys\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665692    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-etc-kubernetes\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665715    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-device-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cnibin\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.666514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-tmp-dir\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-etc-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-host\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665827    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-cni-bin\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh9vf\" (UniqueName: \"kubernetes.io/projected/0cf6192d-d2e9-498b-b582-32305e1572d7-kube-api-access-qh9vf\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665903    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-iptables-alerter-script\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665945    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-slash\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665965    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-kubernetes\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.665981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-registration-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-system-cni-dir\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-hosts-file\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-log-socket\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666111    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-run-netns\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666141    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/259d31ef-0ef7-4655-9121-88fd07468c38-kubelet-config\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666168    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-cni-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.667311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666189    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-conf-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.668231 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666215    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysctl-d\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.668231 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666235    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-run\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.668231 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8bmf\" (UniqueName: \"kubernetes.io/projected/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-kube-api-access-p8bmf\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.668231 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666279    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0162c6de-30e1-4e0b-949e-43b813b1e953-cni-binary-copy\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.668231 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-socket-dir-parent\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.668231 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666311    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-hostroot\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.668231 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666338    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.668231 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666369    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxbqz\" (UniqueName: \"kubernetes.io/projected/1827f1a3-a7e6-4d57-907a-763220ce3e0e-kube-api-access-gxbqz\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.668231 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.666394    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8dqb\" (UniqueName: \"kubernetes.io/projected/78221f6f-2223-4ac2-92da-b763ff2a6b39-kube-api-access-p8dqb\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:24.697540 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697256    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 12:26:24.697540 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697507    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:26:24.697782 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697587    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 12:26:24.697782 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697599    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vg52x\""
Nov 02 12:26:24.697782 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697632    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 12:26:24.697782 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697649    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 12:26:24.697782 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697665    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 12:26:24.697782 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697695    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 12:26:24.697782 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697710    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 12:26:24.697782 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697735    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bbkt9\""
Nov 02 12:26:24.697782 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697777    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 12:26:24.698253 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697835    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-scnn5\""
Nov 02 12:26:24.698253 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.697867    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fhvpr\""
Nov 02 12:26:24.769714 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.769672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-systemd\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.769714 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.769724    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-var-lib-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.770010 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.769985    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-systemd\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.770170 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770052    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-var-lib-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.770170 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-lib-modules\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.770170 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770149    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgpqc\" (UniqueName: \"kubernetes.io/projected/d59c905e-0042-4777-a2fe-d5d5ae04e47d-kube-api-access-vgpqc\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:26:24.770343 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770178    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-system-cni-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.770343 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-cni-multus\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.770343 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770221    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-kubelet\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.770343 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-ovn\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.770343 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770281    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysconfig\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.770601 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770355    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysconfig\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.770601 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770507    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-system-cni-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.770601 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770552    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-cni-multus\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.770601 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770592    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-kubelet\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.770807 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.769922    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-lib-modules\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.770807 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770673    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-kubelet\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.770807 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-sys-fs\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.770807 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.770807 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovnkube-script-lib\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.770807 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.770791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97wkz\" (UniqueName: \"kubernetes.io/projected/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-kube-api-access-97wkz\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.771149 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771036    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/259d31ef-0ef7-4655-9121-88fd07468c38-dbus\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:24.771199 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771144    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-ovn\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.771199 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771192    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/259d31ef-0ef7-4655-9121-88fd07468c38-dbus\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:24.771306 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-tmp\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.771306 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-k8s-cni-cncf-io\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.771408 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771346    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-netns\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.771531 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-netns\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.771655 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771586    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-host-slash\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:26:24.771655 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771632    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-cni-bin\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.771776 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.771776 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d59c905e-0042-4777-a2fe-d5d5ae04e47d-serviceca\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:26:24.771776 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771721    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 12:26:24.771776 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771763    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-kubelet\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.771952 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-cnibin\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.771952 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771860    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-cnibin\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.771952 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771887    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.771952 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmb9\" (UniqueName: \"kubernetes.io/projected/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-kube-api-access-wvmb9\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:26:24.771952 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771936    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-cni-netd\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.772179 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-var-lib-kubelet\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.772179 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovnkube-config\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.772179 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.771995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovn-node-metrics-cert\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.772887 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.772863    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-k8s-cni-cncf-io\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.772973 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.772903    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d-agent-certs\") pod \"konnectivity-agent-26p8v\" (UID: \"83dcdc7a-f1ba-4570-bed5-a40b0d3b213d\") " pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:26:24.772973 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.772941    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-modprobe-d\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.772973 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.772966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-sys\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.773099 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.772990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-etc-kubernetes\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.773099 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-device-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.773099 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773027    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cnibin\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.773099 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-tmp-dir\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:26:24.773099 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-etc-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.773250 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773130    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-etc-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.773250 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773226    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-modprobe-d\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.773311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773266    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-sys\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.773311 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773300    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-etc-kubernetes\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.773374 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773352    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-device-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.773429 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773398    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773763    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-host-slash\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773776    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovnkube-script-lib\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773820    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-host\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773829    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-cni-bin\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-cni-bin\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773881    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d59c905e-0042-4777-a2fe-d5d5ae04e47d-serviceca\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773878    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773925    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-cni-bin\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773940    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773944    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-host\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9vf\" (UniqueName: \"kubernetes.io/projected/0cf6192d-d2e9-498b-b582-32305e1572d7-kube-api-access-qh9vf\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773971    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cnibin\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-iptables-alerter-script\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.773996    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-cni-netd\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-slash\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.774056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774043    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-kubernetes\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-registration-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774110    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-system-cni-dir\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-hosts-file\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774160    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-log-socket\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774174    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-run-netns\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/259d31ef-0ef7-4655-9121-88fd07468c38-kubelet-config\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-cni-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-conf-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysctl-d\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-run\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8bmf\" (UniqueName: \"kubernetes.io/projected/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-kube-api-access-p8bmf\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0162c6de-30e1-4e0b-949e-43b813b1e953-cni-binary-copy\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-socket-dir-parent\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-hostroot\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774460    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.774982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxbqz\" (UniqueName: \"kubernetes.io/projected/1827f1a3-a7e6-4d57-907a-763220ce3e0e-kube-api-access-gxbqz\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774521    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8dqb\" (UniqueName: \"kubernetes.io/projected/78221f6f-2223-4ac2-92da-b763ff2a6b39-kube-api-access-p8dqb\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774549    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-systemd-units\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774621    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-tuned\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774632    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovnkube-config\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-multus-certs\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-socket-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774698    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-var-lib-kubelet\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774744    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdjq\" (UniqueName: \"kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq\") pod \"network-check-target-2hzq7\" (UID: \"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60\") " pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-node-log\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysctl-conf\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774860    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d-konnectivity-ca\") pod \"konnectivity-agent-26p8v\" (UID: \"83dcdc7a-f1ba-4570-bed5-a40b0d3b213d\") " pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774921    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-sys-fs\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774964    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-systemd\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.775837 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.774995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv5cq\" (UniqueName: \"kubernetes.io/projected/0162c6de-30e1-4e0b-949e-43b813b1e953-kube-api-access-pv5cq\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.775025    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-daemon-config\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.775056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5x28\" (UniqueName: \"kubernetes.io/projected/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-kube-api-access-c5x28\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.775103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.775170    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.775292    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-iptables-alerter-script\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.775368    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.775441    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d-konnectivity-ca\") pod \"konnectivity-agent-26p8v\" (UID: \"83dcdc7a-f1ba-4570-bed5-a40b0d3b213d\") " pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:24.775557    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:24.775637    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret podName:259d31ef-0ef7-4655-9121-88fd07468c38 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:25.27561647 +0000 UTC m=+3.111998355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret") pod "global-pull-secret-syncer-x7jpb" (UID: "259d31ef-0ef7-4655-9121-88fd07468c38") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.775663    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-tmp-dir\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.775813    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-multus-certs\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.775872    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-socket-dir-parent\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.775918    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-hostroot\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776040    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-socket-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776123    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776148    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-slash\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.776754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776285    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-kubernetes\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776299    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0162c6de-30e1-4e0b-949e-43b813b1e953-cni-binary-copy\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-env-overrides\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d59c905e-0042-4777-a2fe-d5d5ae04e47d-host\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776516    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776588    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-registration-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776632    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-systemd-units\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776668    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776674    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-system-cni-dir\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776715    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d59c905e-0042-4777-a2fe-d5d5ae04e47d-host\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776728    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/259d31ef-0ef7-4655-9121-88fd07468c38-kubelet-config\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776780    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-log-socket\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776796    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-hosts-file\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776787    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-env-overrides\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776854    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-conf-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776891    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-run-netns\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776908    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d-agent-certs\") pod \"konnectivity-agent-26p8v\" (UID: \"83dcdc7a-f1ba-4570-bed5-a40b0d3b213d\") " pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:26:24.777626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.776953    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-cni-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.778227 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.777119    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysctl-d\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.778227 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.777228    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-systemd\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.778227 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.777282    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-run\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.778227 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.777350    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-daemon-config\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.778227 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.777438    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.778227 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.777455    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-tmp\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.778227 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:24.777538    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:24.778227 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:24.777596    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs podName:78221f6f-2223-4ac2-92da-b763ff2a6b39 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:25.277577326 +0000 UTC m=+3.113959203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs") pod "network-metrics-daemon-2vkgz" (UID: "78221f6f-2223-4ac2-92da-b763ff2a6b39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:24.778227 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.777601    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-node-log\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.778227 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.777693    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysctl-conf\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.780128 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.780104    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovn-node-metrics-cert\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.780409 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.780367    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-tuned\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.799024 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.798948    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgpqc\" (UniqueName: \"kubernetes.io/projected/d59c905e-0042-4777-a2fe-d5d5ae04e47d-kube-api-access-vgpqc\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:26:24.799831 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.799798    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97wkz\" (UniqueName: \"kubernetes.io/projected/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-kube-api-access-97wkz\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.806467 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.806435    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9vf\" (UniqueName: \"kubernetes.io/projected/0cf6192d-d2e9-498b-b582-32305e1572d7-kube-api-access-qh9vf\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.806467 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.806463    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmb9\" (UniqueName: \"kubernetes.io/projected/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-kube-api-access-wvmb9\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:26:24.806870 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.806845    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxbqz\" (UniqueName: \"kubernetes.io/projected/1827f1a3-a7e6-4d57-907a-763220ce3e0e-kube-api-access-gxbqz\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.812899 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.812874    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8dqb\" (UniqueName: \"kubernetes.io/projected/78221f6f-2223-4ac2-92da-b763ff2a6b39-kube-api-access-p8dqb\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:24.813298 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.813279    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv5cq\" (UniqueName: \"kubernetes.io/projected/0162c6de-30e1-4e0b-949e-43b813b1e953-kube-api-access-pv5cq\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.814195 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:24.814081    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:24.814195 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:24.814105    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:24.814195 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:24.814120    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fwdjq for pod openshift-network-diagnostics/network-check-target-2hzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:24.814195 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:24.814172    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq podName:18c61308-36a9-4eb4-9cc9-ac85bf1bdd60 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:25.314157773 +0000 UTC m=+3.150539649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fwdjq" (UniqueName: "kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq") pod "network-check-target-2hzq7" (UID: "18c61308-36a9-4eb4-9cc9-ac85bf1bdd60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:24.815972 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.815948    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5x28\" (UniqueName: \"kubernetes.io/projected/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-kube-api-access-c5x28\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:26:24.817088 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.817066    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8bmf\" (UniqueName: \"kubernetes.io/projected/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-kube-api-access-p8bmf\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.885619 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.885528    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5qqx4"
Nov 02 12:26:24.901172 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.901143    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:26:24.901401 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:24.901365    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0162c6de_30e1_4e0b_949e_43b813b1e953.slice/crio-ca3d73812b549dbba9eec19c804667161aa5f128dfb36b0096e84230fe72d876 WatchSource:0}: Error finding container ca3d73812b549dbba9eec19c804667161aa5f128dfb36b0096e84230fe72d876: Status 404 returned error can't find the container with id ca3d73812b549dbba9eec19c804667161aa5f128dfb36b0096e84230fe72d876
Nov 02 12:26:24.912809 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:24.912781    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf6192d_d2e9_498b_b582_32305e1572d7.slice/crio-b8c1a1eb2cfe560549d2d35c957c0c252df53c896f45ae08a1e00315e363a389 WatchSource:0}: Error finding container b8c1a1eb2cfe560549d2d35c957c0c252df53c896f45ae08a1e00315e363a389: Status 404 returned error can't find the container with id b8c1a1eb2cfe560549d2d35c957c0c252df53c896f45ae08a1e00315e363a389
Nov 02 12:26:24.922222 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.922197    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:26:24.924342 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.924316    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:26:24.932596 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.932152    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:24.936037 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:24.936000    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83dcdc7a_f1ba_4570_bed5_a40b0d3b213d.slice/crio-69c6d5dffcb708513d462969fcde74577a883a833d0af76bbadf3a54581c7d81 WatchSource:0}: Error finding container 69c6d5dffcb708513d462969fcde74577a883a833d0af76bbadf3a54581c7d81: Status 404 returned error can't find the container with id 69c6d5dffcb708513d462969fcde74577a883a833d0af76bbadf3a54581c7d81
Nov 02 12:26:24.945531 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.945497    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:26:24.949124 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.949099    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:26:24.955024 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.954981    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:26:24.957773 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:24.957753    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:26:24.960335 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:24.960292    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718bc5cd_de70_4920_9e8f_5bc14fe1f45c.slice/crio-e0bbe92ab67ecef933ae41cd8a41beb6787609485db5da5a19d1dac4c9b9f358 WatchSource:0}: Error finding container e0bbe92ab67ecef933ae41cd8a41beb6787609485db5da5a19d1dac4c9b9f358: Status 404 returned error can't find the container with id e0bbe92ab67ecef933ae41cd8a41beb6787609485db5da5a19d1dac4c9b9f358
Nov 02 12:26:24.964233 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:24.964207    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89c2be7_3500_4d9a_ac82_4ef367b83e3e.slice/crio-cf82277f49d8e3ca224e9eb33c605be11e0445d211276acc34b917b79b268979 WatchSource:0}: Error finding container cf82277f49d8e3ca224e9eb33c605be11e0445d211276acc34b917b79b268979: Status 404 returned error can't find the container with id cf82277f49d8e3ca224e9eb33c605be11e0445d211276acc34b917b79b268979
Nov 02 12:26:24.971494 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:24.971466    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1827f1a3_a7e6_4d57_907a_763220ce3e0e.slice/crio-ba748218a38311cf4bf3a380834741cab72cab6a0a9c885f0a9a23cc6241f87c WatchSource:0}: Error finding container ba748218a38311cf4bf3a380834741cab72cab6a0a9c885f0a9a23cc6241f87c: Status 404 returned error can't find the container with id ba748218a38311cf4bf3a380834741cab72cab6a0a9c885f0a9a23cc6241f87c
Nov 02 12:26:24.972765 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:26:24.972673    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59c905e_0042_4777_a2fe_d5d5ae04e47d.slice/crio-157a29163cc443f325ee6ca2d4bb24477fe7d5c24fad4cb0a6bdbaefb07c52f7 WatchSource:0}: Error finding container 157a29163cc443f325ee6ca2d4bb24477fe7d5c24fad4cb0a6bdbaefb07c52f7: Status 404 returned error can't find the container with id 157a29163cc443f325ee6ca2d4bb24477fe7d5c24fad4cb0a6bdbaefb07c52f7
Nov 02 12:26:25.280403 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.280360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:25.280562 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.280434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:25.280562 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:25.280545    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:25.280665 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:25.280557    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:25.280665 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:25.280626    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret podName:259d31ef-0ef7-4655-9121-88fd07468c38 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:26.280609719 +0000 UTC m=+4.116991583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret") pod "global-pull-secret-syncer-x7jpb" (UID: "259d31ef-0ef7-4655-9121-88fd07468c38") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:25.280665 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:25.280646    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs podName:78221f6f-2223-4ac2-92da-b763ff2a6b39 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:26.280637014 +0000 UTC m=+4.117018874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs") pod "network-metrics-daemon-2vkgz" (UID: "78221f6f-2223-4ac2-92da-b763ff2a6b39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:25.381662 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.381491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdjq\" (UniqueName: \"kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq\") pod \"network-check-target-2hzq7\" (UID: \"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60\") " pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:25.381817 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:25.381793    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:25.381882 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:25.381826    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:25.381882 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:25.381841    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fwdjq for pod openshift-network-diagnostics/network-check-target-2hzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:25.381995 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:25.381908    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq podName:18c61308-36a9-4eb4-9cc9-ac85bf1bdd60 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:26.381887973 +0000 UTC m=+4.218269854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fwdjq" (UniqueName: "kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq") pod "network-check-target-2hzq7" (UID: "18c61308-36a9-4eb4-9cc9-ac85bf1bdd60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:25.615299 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.615224    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 12:21:24 +0000 UTC" deadline="2028-06-27 13:04:22.908344034 +0000 UTC"
Nov 02 12:26:25.615299 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.615293    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14472h37m57.293055446s"
Nov 02 12:26:25.710117 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.710056    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal" event={"ID":"f2aebf6f2caaa3c2095ff6e33664045c","Type":"ContainerStarted","Data":"682862ac54cd5f3423801db9cac745106ed992612a6e7e4dd13aba7fc8c3ef28"}
Nov 02 12:26:25.718872 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.718834    2582 generic.go:356] "Generic (PLEG): container finished" podID="55b1ac54009e3b5ad6df9f2e917c2d8b" containerID="0cab7c354846ab96475eea5db8af5c1fbb57fa56d6e6da995b0abe47ec578798" exitCode=0
Nov 02 12:26:25.719056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.718936    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal" event={"ID":"55b1ac54009e3b5ad6df9f2e917c2d8b","Type":"ContainerDied","Data":"0cab7c354846ab96475eea5db8af5c1fbb57fa56d6e6da995b0abe47ec578798"}
Nov 02 12:26:25.725804 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.725766    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mdzbn" event={"ID":"d59c905e-0042-4777-a2fe-d5d5ae04e47d","Type":"ContainerStarted","Data":"157a29163cc443f325ee6ca2d4bb24477fe7d5c24fad4cb0a6bdbaefb07c52f7"}
Nov 02 12:26:25.738770 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.737326    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k49jr" event={"ID":"b89c2be7-3500-4d9a-ac82-4ef367b83e3e","Type":"ContainerStarted","Data":"cf82277f49d8e3ca224e9eb33c605be11e0445d211276acc34b917b79b268979"}
Nov 02 12:26:25.739506 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.739475    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9w9rr" event={"ID":"718bc5cd-de70-4920-9e8f-5bc14fe1f45c","Type":"ContainerStarted","Data":"e0bbe92ab67ecef933ae41cd8a41beb6787609485db5da5a19d1dac4c9b9f358"}
Nov 02 12:26:25.744569 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.744451    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk" event={"ID":"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471","Type":"ContainerStarted","Data":"f83efbb007fca102cf724127bf3e9d727ccf747a86eddefcd6ff5e20c45adaad"}
Nov 02 12:26:25.757968 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.757555    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c4mfn" event={"ID":"a41b8e21-8e18-4e58-85d9-c24ba888b2f6","Type":"ContainerStarted","Data":"d064abf07fa1a44346d0f42e8579fc1e71d70a91f06f29e04286221680a236d3"}
Nov 02 12:26:25.771375 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.771345    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-26p8v" event={"ID":"83dcdc7a-f1ba-4570-bed5-a40b0d3b213d","Type":"ContainerStarted","Data":"69c6d5dffcb708513d462969fcde74577a883a833d0af76bbadf3a54581c7d81"}
Nov 02 12:26:25.786127 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.786089    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspl8" event={"ID":"1827f1a3-a7e6-4d57-907a-763220ce3e0e","Type":"ContainerStarted","Data":"ba748218a38311cf4bf3a380834741cab72cab6a0a9c885f0a9a23cc6241f87c"}
Nov 02 12:26:25.788074 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.788001    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql" event={"ID":"0cf6192d-d2e9-498b-b582-32305e1572d7","Type":"ContainerStarted","Data":"b8c1a1eb2cfe560549d2d35c957c0c252df53c896f45ae08a1e00315e363a389"}
Nov 02 12:26:25.803773 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.803727    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5qqx4" event={"ID":"0162c6de-30e1-4e0b-949e-43b813b1e953","Type":"ContainerStarted","Data":"ca3d73812b549dbba9eec19c804667161aa5f128dfb36b0096e84230fe72d876"}
Nov 02 12:26:25.811282 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:25.811226    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal" podStartSLOduration=1.811207926 podStartE2EDuration="1.811207926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:25.75772286 +0000 UTC m=+3.594104771" watchObservedRunningTime="2026-11-02 12:26:25.811207926 +0000 UTC m=+3.647589810"
Nov 02 12:26:26.291267 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:26.291219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:26.291469 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:26.291298    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:26.291534 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:26.291473    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:26.291590 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:26.291548    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs podName:78221f6f-2223-4ac2-92da-b763ff2a6b39 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:28.291527531 +0000 UTC m=+6.127909401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs") pod "network-metrics-daemon-2vkgz" (UID: "78221f6f-2223-4ac2-92da-b763ff2a6b39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:26.292336 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:26.292155    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:26.292336 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:26.292234    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret podName:259d31ef-0ef7-4655-9121-88fd07468c38 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:28.292204434 +0000 UTC m=+6.128586299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret") pod "global-pull-secret-syncer-x7jpb" (UID: "259d31ef-0ef7-4655-9121-88fd07468c38") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:26.392577 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:26.392531    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdjq\" (UniqueName: \"kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq\") pod \"network-check-target-2hzq7\" (UID: \"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60\") " pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:26.392794 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:26.392774    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:26.392859 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:26.392804    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:26.392859 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:26.392821    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fwdjq for pod openshift-network-diagnostics/network-check-target-2hzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:26.392984 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:26.392889    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq podName:18c61308-36a9-4eb4-9cc9-ac85bf1bdd60 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:28.392868197 +0000 UTC m=+6.229250067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fwdjq" (UniqueName: "kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq") pod "network-check-target-2hzq7" (UID: "18c61308-36a9-4eb4-9cc9-ac85bf1bdd60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:26.699111 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:26.696958    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:26.699111 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:26.697101    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:26.699111 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:26.697514    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:26.699111 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:26.697630    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:26.699111 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:26.697695    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:26.699111 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:26.697768    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:26.827710 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:26.827635    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal" event={"ID":"55b1ac54009e3b5ad6df9f2e917c2d8b","Type":"ContainerStarted","Data":"04875313b26af7d60c807628d6c07f10e95f95d802d12d26ee6f1094c3e01202"}
Nov 02 12:26:28.313500 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:28.313273    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:28.313500 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:28.313365    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:28.313500 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:28.313500    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:28.314018 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:28.313570    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret podName:259d31ef-0ef7-4655-9121-88fd07468c38 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:32.313551162 +0000 UTC m=+10.149933037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret") pod "global-pull-secret-syncer-x7jpb" (UID: "259d31ef-0ef7-4655-9121-88fd07468c38") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:28.314018 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:28.313991    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:28.314220 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:28.314047    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs podName:78221f6f-2223-4ac2-92da-b763ff2a6b39 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:32.314030054 +0000 UTC m=+10.150411920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs") pod "network-metrics-daemon-2vkgz" (UID: "78221f6f-2223-4ac2-92da-b763ff2a6b39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:28.414641 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:28.414546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdjq\" (UniqueName: \"kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq\") pod \"network-check-target-2hzq7\" (UID: \"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60\") " pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:28.414815 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:28.414720    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:28.414815 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:28.414736    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:28.414815 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:28.414746    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fwdjq for pod openshift-network-diagnostics/network-check-target-2hzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:28.414815 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:28.414799    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq podName:18c61308-36a9-4eb4-9cc9-ac85bf1bdd60 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:32.414779702 +0000 UTC m=+10.251161562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fwdjq" (UniqueName: "kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq") pod "network-check-target-2hzq7" (UID: "18c61308-36a9-4eb4-9cc9-ac85bf1bdd60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:28.697390 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:28.697085    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:28.697390 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:28.697222    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:28.697806 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:28.697638    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:28.697806 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:28.697758    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:28.697914 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:28.697832    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:28.697948 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:28.697917    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:28.839923 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:28.839877    2582 generic.go:356] "Generic (PLEG): container finished" podID="1827f1a3-a7e6-4d57-907a-763220ce3e0e" containerID="e46ac257fe9cfc77cb6df5163ad486e4b9e81f17356d70104250826bfa67bafb" exitCode=0
Nov 02 12:26:28.840132 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:28.839955    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspl8" event={"ID":"1827f1a3-a7e6-4d57-907a-763220ce3e0e","Type":"ContainerDied","Data":"e46ac257fe9cfc77cb6df5163ad486e4b9e81f17356d70104250826bfa67bafb"}
Nov 02 12:26:28.954760 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:28.954643    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal" podStartSLOduration=4.9546217630000005 podStartE2EDuration="4.954621763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:26.873275929 +0000 UTC m=+4.709657815" watchObservedRunningTime="2026-11-02 12:26:28.954621763 +0000 UTC m=+6.791003647"
Nov 02 12:26:29.855465 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:29.854709    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-26p8v" event={"ID":"83dcdc7a-f1ba-4570-bed5-a40b0d3b213d","Type":"ContainerStarted","Data":"573afcc586a991a9f3aeae619e8a86affef60047a8b66347da0a06583482c136"}
Nov 02 12:26:29.902730 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:29.901891    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-26p8v" podStartSLOduration=2.505460519 podStartE2EDuration="6.90186756s" totalImagesPullingTime="4.396407041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:29.901205384 +0000 UTC m=+7.737587270" watchObservedRunningTime="2026-11-02 12:26:29.90186756 +0000 UTC m=+7.738249446"
Nov 02 12:26:30.696892 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:30.696796    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:30.697079 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:30.696967    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:30.697079 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:30.697071    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:30.697193 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:30.697161    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:30.697335 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:30.697232    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:30.697335 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:30.697313    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:30.864436 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:30.864398    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mdzbn" event={"ID":"d59c905e-0042-4777-a2fe-d5d5ae04e47d","Type":"ContainerStarted","Data":"b66e783ba19c80ae760edb16a5517db9850a62502d3b05ec11b2bb254c5977e2"}
Nov 02 12:26:31.175270 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:31.175229    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:31.486496 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:31.486389    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:31.750071 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:31.749916    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:26:31.871526 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:31.871483    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql" event={"ID":"0cf6192d-d2e9-498b-b582-32305e1572d7","Type":"ContainerStarted","Data":"75664ecda665d96d87b0683f8d3d0c91da6c798b061cdfb4656b531ddd0a4f2e"}
Nov 02 12:26:32.220090 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:32.220001    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:32.352576 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:32.351685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:32.352576 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:32.351762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:32.352576 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:32.351925    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:32.352576 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:32.351998    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs podName:78221f6f-2223-4ac2-92da-b763ff2a6b39 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:40.351977688 +0000 UTC m=+18.188359572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs") pod "network-metrics-daemon-2vkgz" (UID: "78221f6f-2223-4ac2-92da-b763ff2a6b39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:32.352576 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:32.352458    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:32.352576 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:32.352518    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret podName:259d31ef-0ef7-4655-9121-88fd07468c38 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:40.352500686 +0000 UTC m=+18.188882577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret") pod "global-pull-secret-syncer-x7jpb" (UID: "259d31ef-0ef7-4655-9121-88fd07468c38") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:32.356135 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:32.355997    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:32.452805 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:32.452762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdjq\" (UniqueName: \"kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq\") pod \"network-check-target-2hzq7\" (UID: \"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60\") " pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:32.453007 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:32.452988    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:32.453098 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:32.453009    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:32.453098 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:32.453023    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fwdjq for pod openshift-network-diagnostics/network-check-target-2hzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:32.453220 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:32.453112    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq podName:18c61308-36a9-4eb4-9cc9-ac85bf1bdd60 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:40.453091804 +0000 UTC m=+18.289473692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fwdjq" (UniqueName: "kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq") pod "network-check-target-2hzq7" (UID: "18c61308-36a9-4eb4-9cc9-ac85bf1bdd60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:32.697572 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:32.697529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:32.697763 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:32.697690    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:32.697763 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:32.697762    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:32.697880 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:32.697847    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:32.697938 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:32.697891    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:32.697993 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:32.697957    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:32.806657 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:32.806619    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 12:26:32.877713 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:32.877669    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql" event={"ID":"0cf6192d-d2e9-498b-b582-32305e1572d7","Type":"ContainerStarted","Data":"20ece056a714381869a5abc8836ca7dcc64348209ede21ebb363e65abb0f19ce"}
Nov 02 12:26:33.627243 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:33.626394    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T12:26:32.806640917Z","UUID":"638e047b-b567-42f6-9d07-e2e973cfad97","Handler":null,"Name":"","Endpoint":""}
Nov 02 12:26:33.628723 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:33.628578    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 12:26:33.628723 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:33.628616    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 12:26:33.882296 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:33.881983    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k49jr" event={"ID":"b89c2be7-3500-4d9a-ac82-4ef367b83e3e","Type":"ContainerStarted","Data":"3e7a437f40758da8d07db39d9d6af470ba0795b505344dde934d8ad3ae6c1f2f"}
Nov 02 12:26:33.884808 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:33.884774    2582 generic.go:356] "Generic (PLEG): container finished" podID="1827f1a3-a7e6-4d57-907a-763220ce3e0e" containerID="9b0a6f1d53dfd70d04bd5b3a4611244d916ca297d6941b23805c2f447f62234d" exitCode=0
Nov 02 12:26:33.884957 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:33.884850    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspl8" event={"ID":"1827f1a3-a7e6-4d57-907a-763220ce3e0e","Type":"ContainerDied","Data":"9b0a6f1d53dfd70d04bd5b3a4611244d916ca297d6941b23805c2f447f62234d"}
Nov 02 12:26:33.888296 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:33.888265    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5qqx4" event={"ID":"0162c6de-30e1-4e0b-949e-43b813b1e953","Type":"ContainerStarted","Data":"1725d52df59a493f6790195d33852260fcc9a04d6757ddface6116cdec30cada"}
Nov 02 12:26:33.928631 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:33.928577    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-k49jr" podStartSLOduration=2.421442085 podStartE2EDuration="10.928559076s" totalImagesPullingTime="8.507116991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:33.927545824 +0000 UTC m=+11.763927707" watchObservedRunningTime="2026-11-02 12:26:33.928559076 +0000 UTC m=+11.764940959"
Nov 02 12:26:33.928775 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:33.928679    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mdzbn" podStartSLOduration=5.831283249 podStartE2EDuration="10.928670822s" totalImagesPullingTime="5.097387573s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:30.910435781 +0000 UTC m=+8.746817666" watchObservedRunningTime="2026-11-02 12:26:33.928670822 +0000 UTC m=+11.765052708"
Nov 02 12:26:33.976510 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:33.976422    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5qqx4" podStartSLOduration=2.500872192 podStartE2EDuration="10.976406416s" totalImagesPullingTime="8.475534224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:33.975811033 +0000 UTC m=+11.812192931" watchObservedRunningTime="2026-11-02 12:26:33.976406416 +0000 UTC m=+11.812788299"
Nov 02 12:26:34.284966 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.284872    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:26:34.286189 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.286167    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:26:34.701232 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.701019    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:34.701232 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.701078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:34.701232 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.701034    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:34.701232 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:34.701186    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:34.701518 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:34.701268    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:34.701518 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:34.701351    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:34.892709 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.892623    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c4mfn" event={"ID":"a41b8e21-8e18-4e58-85d9-c24ba888b2f6","Type":"ContainerStarted","Data":"bd36bf2b30eeaacf5caf9fa83e7e20aa98dd555c9048d2d3935cca0d0745191f"}
Nov 02 12:26:34.895404 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.895372    2582 generic.go:356] "Generic (PLEG): container finished" podID="1827f1a3-a7e6-4d57-907a-763220ce3e0e" containerID="039d443308fceef2ae8f242348a0e91a4fbb695a384a5ad2142d30f524d8e5d5" exitCode=0
Nov 02 12:26:34.895404 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.895423    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspl8" event={"ID":"1827f1a3-a7e6-4d57-907a-763220ce3e0e","Type":"ContainerDied","Data":"039d443308fceef2ae8f242348a0e91a4fbb695a384a5ad2142d30f524d8e5d5"}
Nov 02 12:26:34.899061 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.899038    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql" event={"ID":"0cf6192d-d2e9-498b-b582-32305e1572d7","Type":"ContainerStarted","Data":"42f4fa5482ae8745f7fcbc1781408d9c335ed320e5b6bca59d9a079a0caa128d"}
Nov 02 12:26:34.900014 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.899982    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:26:34.919480 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.919453    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:34.934092 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.934050    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c4mfn" podStartSLOduration=3.650314727 podStartE2EDuration="11.934037407s" totalImagesPullingTime="8.28372268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:34.933927425 +0000 UTC m=+12.770309306" watchObservedRunningTime="2026-11-02 12:26:34.934037407 +0000 UTC m=+12.770419287"
Nov 02 12:26:34.981123 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:34.981065    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql" podStartSLOduration=3.3769833719999998 podStartE2EDuration="11.981045953s" totalImagesPullingTime="8.604062581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:34.981026949 +0000 UTC m=+12.817408829" watchObservedRunningTime="2026-11-02 12:26:34.981045953 +0000 UTC m=+12.817427827"
Nov 02 12:26:35.043267 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:35.043227    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:35.168502 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:35.168424    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:35.902817 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:35.902783    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9w9rr" event={"ID":"718bc5cd-de70-4920-9e8f-5bc14fe1f45c","Type":"ContainerStarted","Data":"e459464613e579cd9f49ecc5674213a25fd2c7ee10d6da01d5c4ba4ae65ef7b7"}
Nov 02 12:26:35.905987 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:35.905957    2582 generic.go:356] "Generic (PLEG): container finished" podID="1827f1a3-a7e6-4d57-907a-763220ce3e0e" containerID="2504f59d39bceafc59e33532879d30dfc710104740c4425e1b4e66b50e522510" exitCode=0
Nov 02 12:26:35.906117 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:35.906039    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspl8" event={"ID":"1827f1a3-a7e6-4d57-907a-763220ce3e0e","Type":"ContainerDied","Data":"2504f59d39bceafc59e33532879d30dfc710104740c4425e1b4e66b50e522510"}
Nov 02 12:26:35.947955 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:35.947904    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9w9rr" podStartSLOduration=2.174660439 podStartE2EDuration="12.947891385s" totalImagesPullingTime="10.773230946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:35.947223979 +0000 UTC m=+13.783605859" watchObservedRunningTime="2026-11-02 12:26:35.947891385 +0000 UTC m=+13.784273262"
Nov 02 12:26:36.696558 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:36.696527    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:36.696714 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:36.696653    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:36.697080 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:36.697062    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:36.697174 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:36.697160    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:36.697219 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:36.697209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:36.697282 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:36.697263    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:37.555958 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:37.555926    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:37.673295 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:37.673258    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:37.914267 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:37.914222    2582 generic.go:356] "Generic (PLEG): container finished" podID="1827f1a3-a7e6-4d57-907a-763220ce3e0e" containerID="b9e91cc4b5d14290085fc725e1e5226eac7e09114688b928e8908e8e651ba1a2" exitCode=0
Nov 02 12:26:37.914267 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:37.914271    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspl8" event={"ID":"1827f1a3-a7e6-4d57-907a-763220ce3e0e","Type":"ContainerDied","Data":"b9e91cc4b5d14290085fc725e1e5226eac7e09114688b928e8908e8e651ba1a2"}
Nov 02 12:26:38.390437 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:38.390127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:38.520437 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:38.520405    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:38.686580 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:38.686473    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:38.696603 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:38.696579    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:38.696770 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:38.696653    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:38.696770 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:38.696686    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:38.696886 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:38.696798    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:38.696937 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:38.696871    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:38.697023 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:38.696982    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:38.798474 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:38.798437    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:38.920322 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:38.920285    2582 generic.go:356] "Generic (PLEG): container finished" podID="1827f1a3-a7e6-4d57-907a-763220ce3e0e" containerID="6a71f99776bf44387c17651af19a6d32c358d439ffc5b12d53e97d9f291079b5" exitCode=0
Nov 02 12:26:38.920485 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:38.920332    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspl8" event={"ID":"1827f1a3-a7e6-4d57-907a-763220ce3e0e","Type":"ContainerDied","Data":"6a71f99776bf44387c17651af19a6d32c358d439ffc5b12d53e97d9f291079b5"}
Nov 02 12:26:39.496457 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:39.496411    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:39.631720 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:39.631688    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:39.926321 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:39.926251    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fspl8" event={"ID":"1827f1a3-a7e6-4d57-907a-763220ce3e0e","Type":"ContainerStarted","Data":"efb30940fc39fa71fc75023867372bc58b5628eb9c964d04421ce607b6c377ef"}
Nov 02 12:26:40.003552 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:40.003498    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fspl8" podStartSLOduration=6.529538192 podStartE2EDuration="17.003482766s" totalImagesPullingTime="10.473944574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:39.989446611 +0000 UTC m=+17.825828491" watchObservedRunningTime="2026-11-02 12:26:40.003482766 +0000 UTC m=+17.839864645"
Nov 02 12:26:40.126800 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:40.126767    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:40.416349 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:40.416256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:40.416349 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:40.416321    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:40.416533 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:40.416407    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:40.416533 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:40.416407    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:40.416533 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:40.416461    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret podName:259d31ef-0ef7-4655-9121-88fd07468c38 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:56.416447453 +0000 UTC m=+34.252829324 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret") pod "global-pull-secret-syncer-x7jpb" (UID: "259d31ef-0ef7-4655-9121-88fd07468c38") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:40.416533 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:40.416475    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs podName:78221f6f-2223-4ac2-92da-b763ff2a6b39 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:56.416467797 +0000 UTC m=+34.252849655 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs") pod "network-metrics-daemon-2vkgz" (UID: "78221f6f-2223-4ac2-92da-b763ff2a6b39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:40.517646 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:40.517613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdjq\" (UniqueName: \"kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq\") pod \"network-check-target-2hzq7\" (UID: \"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60\") " pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:40.517800 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:40.517750    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:40.517800 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:40.517763    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:40.517800 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:40.517772    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fwdjq for pod openshift-network-diagnostics/network-check-target-2hzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:40.517906 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:40.517820    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq podName:18c61308-36a9-4eb4-9cc9-ac85bf1bdd60 nodeName:}" failed. No retries permitted until 2026-11-02 12:26:56.517808097 +0000 UTC m=+34.354189967 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fwdjq" (UniqueName: "kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq") pod "network-check-target-2hzq7" (UID: "18c61308-36a9-4eb4-9cc9-ac85bf1bdd60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:40.697162 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:40.697138    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:40.697285 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:40.697180    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:40.697349 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:40.697302    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:40.697407 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:40.697371    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:40.697460 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:40.697387    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:40.697460 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:40.697442    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:40.930599 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:40.930433    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk" event={"ID":"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471","Type":"ContainerStarted","Data":"f1fca669041cb66ed63ac44f68e9d752ed752fc978799feb9438c96de2cb7fc5"}
Nov 02 12:26:40.930912 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:40.930641    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk" event={"ID":"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471","Type":"ContainerStarted","Data":"ea8f2d286cd543accc3ba9848d2a477b7a7908d9274ee203d74ae3a3a54d476a"}
Nov 02 12:26:40.930912 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:40.930676    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk" event={"ID":"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471","Type":"ContainerStarted","Data":"c7c51fcbcb8361c16f532f36cd87e09f64088b0563c92aa3021f94552f9fdf7a"}
Nov 02 12:26:40.930912 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:40.930705    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk" event={"ID":"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471","Type":"ContainerStarted","Data":"5c77195edc0379da975cda33b371663f0a947a90aef49a234f9f76ddbe4b0acd"}
Nov 02 12:26:41.935296 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:41.935259    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk" event={"ID":"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471","Type":"ContainerStarted","Data":"cb03434909fc4842374f0f2052c7dfd9ab1565cd7653dd8a5f0e8a9552c8db23"}
Nov 02 12:26:41.935682 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:41.935342    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk" event={"ID":"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471","Type":"ContainerStarted","Data":"13e687924c07aa6cec2cc0f24072914359ad14da56c99cf3e68d61c4ff98923e"}
Nov 02 12:26:42.697834 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:42.697802    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:42.698016 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:42.697940    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:42.698016 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:42.697978    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:42.698117 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:42.698096    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:42.698159 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:42.698132    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:42.698215 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:42.698200    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:42.926431 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:42.926398    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:43.595344 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:43.595316    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:43.942604 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:43.942528    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk" event={"ID":"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471","Type":"ContainerStarted","Data":"b729292e9d859bf0b0359fda72dc676607ccdf2eb3f68be442e90033eccd1479"}
Nov 02 12:26:44.437199 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:44.437110    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:44.572755 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:44.572720    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:44.692510 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:44.692425    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:44.696568 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:44.696552    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:44.696629 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:44.696574    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:44.696678 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:44.696553    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:44.696717 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:44.696644    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:44.696759 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:44.696723    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:44.696813 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:44.696797    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:44.723601 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:44.723571    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k49jr_b89c2be7-3500-4d9a-ac82-4ef367b83e3e/dns-node-resolver/0.log"
Nov 02 12:26:44.878820 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:44.878788    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:45.111537 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:45.111457    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:45.236590 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:45.236553    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:45.723898 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:45.723873    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mdzbn_d59c905e-0042-4777-a2fe-d5d5ae04e47d/node-ca/0.log"
Nov 02 12:26:45.905740 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:45.905709    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:45.952494 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:45.952464    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk" event={"ID":"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471","Type":"ContainerStarted","Data":"74742305fd3689c675220d006d3dd5093c8e625b5530bc0bdb14a7dfe299ca16"}
Nov 02 12:26:45.952685 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:45.952659    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:45.952749 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:45.952730    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:45.967947 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:45.967923    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:45.968135 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:45.968056    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:46.028649 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:46.028560    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:46.048133 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:46.048083    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk" podStartSLOduration=7.319089085 podStartE2EDuration="23.048070053s" totalImagesPullingTime="15.728980968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:26:46.033688904 +0000 UTC m=+23.870070783" watchObservedRunningTime="2026-11-02 12:26:46.048070053 +0000 UTC m=+23.884451932"
Nov 02 12:26:46.301449 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:46.301364    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:46.504164 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:46.504127    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:46.696772 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:46.696689    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:46.696923 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:46.696690    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:46.696923 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:46.696817    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:46.696923 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:46.696697    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:46.696923 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:46.696903    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:46.697103 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:46.696968    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:46.754831 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:46.754802    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:46.954681 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:46.954611    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:47.110031 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:47.109977    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:47.226440 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:47.226366    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:47.709165 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:47.709083    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:47.957015 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:47.956968    2582 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:26:48.298161 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:48.298126    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:48.504097 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:48.504063    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:48.696845 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:48.696766    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:48.696845 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:48.696767    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:48.697088 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:48.696858    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:48.697088 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:48.696967    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:48.697088 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:48.697024    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:48.697187 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:48.697120    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:48.743044 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:48.743012    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:48.896984 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:48.896949    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:49.103270 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:49.103179    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:49.633289 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:49.633243    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:26:49.715184 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:49.715148    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:49.839879 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:49.839839    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:49.970505 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:49.970412    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:50.207491 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:50.207462    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:50.334248 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:50.334167    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:50.500599 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:50.500564    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:50.696507 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:50.696425    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:50.696507 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:50.696441    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:50.696507 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:50.696454    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:50.696747 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:50.696538    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:50.696747 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:50.696597    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:50.696747 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:50.696673    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:50.906878 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:50.906849    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:51.116397 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:51.116359    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:51.704194 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:51.704169    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:52.305685 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:52.305655    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:52.507380 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:52.507346    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:52.697287 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:52.697206    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:52.697448 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:52.697323    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:52.697448 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:52.697338    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:52.697448 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:52.697364    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:52.697448 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:52.697409    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:52.697638 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:52.697460    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:52.896718 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:52.896686    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:53.110944 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:53.110872    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:53.700796 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:53.700764    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:53.905957 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:53.905927    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:54.302512 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:54.302483    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:54.511667 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:54.511628    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:54.699393 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:54.699326    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:54.699521 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:54.699326    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:54.699521 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:54.699427    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:54.699521 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:54.699326    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:54.699611 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:54.699485    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:54.699611 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:54.699585    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:54.903914 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:54.903885    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:55.098533 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:55.098454    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:55.302866 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:55.302831    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:56.436545 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:56.436509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:56.437028 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:56.436553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:56.437028 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:56.436648    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:56.437028 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:56.436657    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:56.437028 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:56.436697    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs podName:78221f6f-2223-4ac2-92da-b763ff2a6b39 nodeName:}" failed. No retries permitted until 2026-11-02 12:27:28.436685001 +0000 UTC m=+66.273066860 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs") pod "network-metrics-daemon-2vkgz" (UID: "78221f6f-2223-4ac2-92da-b763ff2a6b39") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 12:26:56.437028 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:56.436722    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret podName:259d31ef-0ef7-4655-9121-88fd07468c38 nodeName:}" failed. No retries permitted until 2026-11-02 12:27:28.436701575 +0000 UTC m=+66.273083448 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret") pod "global-pull-secret-syncer-x7jpb" (UID: "259d31ef-0ef7-4655-9121-88fd07468c38") : object "kube-system"/"original-pull-secret" not registered
Nov 02 12:26:56.536839 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:56.536810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdjq\" (UniqueName: \"kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq\") pod \"network-check-target-2hzq7\" (UID: \"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60\") " pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:56.537026 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:56.536977    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 12:26:56.537026 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:56.537008    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 12:26:56.537026 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:56.537018    2582 projected.go:196] Error preparing data for projected volume kube-api-access-fwdjq for pod openshift-network-diagnostics/network-check-target-2hzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:56.537139 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:56.537062    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq podName:18c61308-36a9-4eb4-9cc9-ac85bf1bdd60 nodeName:}" failed. No retries permitted until 2026-11-02 12:27:28.537050256 +0000 UTC m=+66.373432118 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fwdjq" (UniqueName: "kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq") pod "network-check-target-2hzq7" (UID: "18c61308-36a9-4eb4-9cc9-ac85bf1bdd60") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 12:26:56.696684 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:56.696608    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:56.696684 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:56.696645    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:56.696868 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:56.696608    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:56.696868 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:56.696714    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:56.696868 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:56.696803    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:56.696958 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:56.696884    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:57.654573 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:57.654538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:57.778137 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:57.778099    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:26:58.696649 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:58.696460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:58.697133 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:58.696460    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:58.697133 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:58.696722    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:26:58.697133 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:58.696841    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:58.697133 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:58.696520    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:58.697133 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:58.696945    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:59.299457 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:59.299430    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x7jpb"]
Nov 02 12:26:59.299623 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:59.299535    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:26:59.299698 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:59.299627    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:26:59.301054 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:59.301025    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2hzq7"]
Nov 02 12:26:59.301264 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:59.301230    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2vkgz"]
Nov 02 12:26:59.301357 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:59.301330    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:26:59.301457 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:59.301433    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:26:59.301744 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:26:59.301727    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:26:59.301927 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:26:59.301903    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:27:00.318576 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:00.318540    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:00.438263 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:00.438236    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:00.699831 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:00.699800    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:27:00.700124 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:00.699804    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:27:00.700124 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:27:00.699915    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:27:00.700124 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:27:00.699966    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:27:01.696883 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:01.696850    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:01.697365 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:27:01.696970    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:27:02.697626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:02.697589    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:27:02.698111 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:27:02.697674    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2hzq7" podUID="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60"
Nov 02 12:27:02.698111 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:02.697769    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:27:02.698111 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:27:02.697858    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2vkgz" podUID="78221f6f-2223-4ac2-92da-b763ff2a6b39"
Nov 02 12:27:03.156198 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:03.156159    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:03.697008 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:03.696967    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:03.697193 ip-10-0-134-229 kubenswrapper[2582]: E1102 12:27:03.697113    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x7jpb" podUID="259d31ef-0ef7-4655-9121-88fd07468c38"
Nov 02 12:27:03.924158 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:03.924125    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeReady"
Nov 02 12:27:03.924630 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:03.924296    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 12:27:03.989975 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:03.989890    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-snnjx"]
Nov 02 12:27:04.012056 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.012027    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nkq67"]
Nov 02 12:27:04.012202 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.012116    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.021940 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.021907    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8mqtc"]
Nov 02 12:27:04.024642 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.024619    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nkq67"
Nov 02 12:27:04.028810 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.028784    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 12:27:04.028945 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.028913    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 12:27:04.029030 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.028980    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rbwvz\""
Nov 02 12:27:04.033115 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.033092    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-snnjx"]
Nov 02 12:27:04.033210 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.033135    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nkq67"]
Nov 02 12:27:04.033262 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.033203    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.041456 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.041436    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tlsnx\""
Nov 02 12:27:04.041639 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.041623    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 12:27:04.041698 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.041623    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 12:27:04.041698 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.041627    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 12:27:04.049917 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.049897    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8mqtc"]
Nov 02 12:27:04.129838 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.129809    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 12:27:04.130098 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.130079    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 12:27:04.130175 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.130079    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 12:27:04.130175 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.130125    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 12:27:04.130175 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.130142    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7sfkq\""
Nov 02 12:27:04.188836 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.188796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-crio-socket\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.188836 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.188835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9b72267e-e7f7-4115-af3e-14a382e659af-config-volume\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.189060 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.188854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.189060 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.188918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.189060 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.189010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b72267e-e7f7-4115-af3e-14a382e659af-metrics-tls\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.189060 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.189045    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p26c\" (UniqueName: \"kubernetes.io/projected/4c8202fc-395e-4188-af53-19a2d40a55fe-kube-api-access-5p26c\") pod \"ingress-canary-nkq67\" (UID: \"4c8202fc-395e-4188-af53-19a2d40a55fe\") " pod="openshift-ingress-canary/ingress-canary-nkq67"
Nov 02 12:27:04.189173 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.189071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnwn9\" (UniqueName: \"kubernetes.io/projected/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-kube-api-access-xnwn9\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.189173 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.189091    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csd2s\" (UniqueName: \"kubernetes.io/projected/9b72267e-e7f7-4115-af3e-14a382e659af-kube-api-access-csd2s\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.189173 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.189169    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b72267e-e7f7-4115-af3e-14a382e659af-tmp-dir\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.189263 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.189185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c8202fc-395e-4188-af53-19a2d40a55fe-cert\") pod \"ingress-canary-nkq67\" (UID: \"4c8202fc-395e-4188-af53-19a2d40a55fe\") " pod="openshift-ingress-canary/ingress-canary-nkq67"
Nov 02 12:27:04.189263 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.189201    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-data-volume\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.290029 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.289943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csd2s\" (UniqueName: \"kubernetes.io/projected/9b72267e-e7f7-4115-af3e-14a382e659af-kube-api-access-csd2s\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.290179 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b72267e-e7f7-4115-af3e-14a382e659af-tmp-dir\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.290179 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290058    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c8202fc-395e-4188-af53-19a2d40a55fe-cert\") pod \"ingress-canary-nkq67\" (UID: \"4c8202fc-395e-4188-af53-19a2d40a55fe\") " pod="openshift-ingress-canary/ingress-canary-nkq67"
Nov 02 12:27:04.290179 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290080    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-data-volume\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.290179 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290112    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-crio-socket\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.290179 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9b72267e-e7f7-4115-af3e-14a382e659af-config-volume\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.290179 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.290179 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290173    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.290500 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b72267e-e7f7-4115-af3e-14a382e659af-metrics-tls\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.290500 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p26c\" (UniqueName: \"kubernetes.io/projected/4c8202fc-395e-4188-af53-19a2d40a55fe-kube-api-access-5p26c\") pod \"ingress-canary-nkq67\" (UID: \"4c8202fc-395e-4188-af53-19a2d40a55fe\") " pod="openshift-ingress-canary/ingress-canary-nkq67"
Nov 02 12:27:04.290500 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnwn9\" (UniqueName: \"kubernetes.io/projected/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-kube-api-access-xnwn9\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.290500 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290348    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-crio-socket\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.290500 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290478    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b72267e-e7f7-4115-af3e-14a382e659af-tmp-dir\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.290730 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290507    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-data-volume\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.290913 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.290894    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9b72267e-e7f7-4115-af3e-14a382e659af-config-volume\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.291037 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.291017    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.294358 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.294334    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.294456 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.294345    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b72267e-e7f7-4115-af3e-14a382e659af-metrics-tls\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.294456 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.294403    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c8202fc-395e-4188-af53-19a2d40a55fe-cert\") pod \"ingress-canary-nkq67\" (UID: \"4c8202fc-395e-4188-af53-19a2d40a55fe\") " pod="openshift-ingress-canary/ingress-canary-nkq67"
Nov 02 12:27:04.315379 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.315355    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csd2s\" (UniqueName: \"kubernetes.io/projected/9b72267e-e7f7-4115-af3e-14a382e659af-kube-api-access-csd2s\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.315504 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.315440    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnwn9\" (UniqueName: \"kubernetes.io/projected/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-kube-api-access-xnwn9\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.315641 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.315626    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p26c\" (UniqueName: \"kubernetes.io/projected/4c8202fc-395e-4188-af53-19a2d40a55fe-kube-api-access-5p26c\") pod \"ingress-canary-nkq67\" (UID: \"4c8202fc-395e-4188-af53-19a2d40a55fe\") " pod="openshift-ingress-canary/ingress-canary-nkq67"
Nov 02 12:27:04.352255 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.352218    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:04.390206 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.390055    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nkq67"
Nov 02 12:27:04.415424 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.415399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:04.696553 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.696524    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:27:04.696710 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.696535    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:27:04.713273 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.713237    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l6cfl\""
Nov 02 12:27:04.713456 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.713295    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 12:27:04.713456 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.713294    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2lqbm\""
Nov 02 12:27:04.713456 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.713334    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 12:27:04.713456 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.713359    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 12:27:04.732821 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.732800    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-snnjx"]
Nov 02 12:27:04.735240 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:27:04.735217    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b72267e_e7f7_4115_af3e_14a382e659af.slice/crio-d7ceb6da9aa7950150360aef73ffba03976248108c7724d3f6320157a3b2a531 WatchSource:0}: Error finding container d7ceb6da9aa7950150360aef73ffba03976248108c7724d3f6320157a3b2a531: Status 404 returned error can't find the container with id d7ceb6da9aa7950150360aef73ffba03976248108c7724d3f6320157a3b2a531
Nov 02 12:27:04.743228 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.743206    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nkq67"]
Nov 02 12:27:04.745466 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:27:04.745444    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8202fc_395e_4188_af53_19a2d40a55fe.slice/crio-5927fee2edddf17965fe3e93cea6caa0de3abc9a67344b2da5770977d8c334d7 WatchSource:0}: Error finding container 5927fee2edddf17965fe3e93cea6caa0de3abc9a67344b2da5770977d8c334d7: Status 404 returned error can't find the container with id 5927fee2edddf17965fe3e93cea6caa0de3abc9a67344b2da5770977d8c334d7
Nov 02 12:27:04.770483 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.770405    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8mqtc"]
Nov 02 12:27:04.771380 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:27:04.771359    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f93fec2_d1e5_4efc_82c6_b029ce9e3d7a.slice/crio-de9c9447ce3611f1658dd55296dfade31452a41ea352a1d821aafda26255bb78 WatchSource:0}: Error finding container de9c9447ce3611f1658dd55296dfade31452a41ea352a1d821aafda26255bb78: Status 404 returned error can't find the container with id de9c9447ce3611f1658dd55296dfade31452a41ea352a1d821aafda26255bb78
Nov 02 12:27:04.773799 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.773768    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:04.915797 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.915754    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:04.993013 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.992909    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mqtc" event={"ID":"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a","Type":"ContainerStarted","Data":"cdaf8a91e55aead1bf7449a6f1b6fc658eb5d9e491b7092b08ac3dc85f47857c"}
Nov 02 12:27:04.993013 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.992968    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mqtc" event={"ID":"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a","Type":"ContainerStarted","Data":"de9c9447ce3611f1658dd55296dfade31452a41ea352a1d821aafda26255bb78"}
Nov 02 12:27:04.994135 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.994116    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nkq67" event={"ID":"4c8202fc-395e-4188-af53-19a2d40a55fe","Type":"ContainerStarted","Data":"5927fee2edddf17965fe3e93cea6caa0de3abc9a67344b2da5770977d8c334d7"}
Nov 02 12:27:04.995203 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:04.995183    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-snnjx" event={"ID":"9b72267e-e7f7-4115-af3e-14a382e659af","Type":"ContainerStarted","Data":"d7ceb6da9aa7950150360aef73ffba03976248108c7724d3f6320157a3b2a531"}
Nov 02 12:27:05.023397 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.023367    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"]
Nov 02 12:27:05.034495 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.034471    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.049645 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.049586    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"]
Nov 02 12:27:05.056830 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.056804    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 12:27:05.056830 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.056804    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 12:27:05.057145 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.056907    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 12:27:05.057145 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.056931    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4xfxz\""
Nov 02 12:27:05.057145 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.056946    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 12:27:05.057145 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.056985    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 12:27:05.058045 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.058016    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:05.066394 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.066371    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lbm4n"]
Nov 02 12:27:05.078147 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.078124    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"]
Nov 02 12:27:05.078268 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.078254    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.090305 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.090281    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.107651 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.107624    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"]
Nov 02 12:27:05.125640 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.125614    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 12:27:05.125796 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.125648    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 12:27:05.125796 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.125646    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 02 12:27:05.125796 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.125726    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-89s57\""
Nov 02 12:27:05.125796 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.125787    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 02 12:27:05.125923 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.125800    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pd4kn\""
Nov 02 12:27:05.156781 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.156745    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 12:27:05.156919 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.156881    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 02 12:27:05.177265 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.177232    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:05.198433 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9623fb-ce84-42fc-aa52-fc263d66ec90-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.198433 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-tls\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.198620 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-sys\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.198620 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198479    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-textfile\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.198620 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198533    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx6p7\" (UniqueName: \"kubernetes.io/projected/d9215ec4-711e-400c-852c-aaaebbd7a7ec-kube-api-access-hx6p7\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.198620 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198603    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.198756 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198646    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-root\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.198756 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.198756 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ff9623fb-ce84-42fc-aa52-fc263d66ec90-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.198756 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198750    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.198882 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.198882 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.198882 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198859    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.198976 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmz9r\" (UniqueName: \"kubernetes.io/projected/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-api-access-jmz9r\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.198976 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198948    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.198976 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198971    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfhm\" (UniqueName: \"kubernetes.io/projected/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-kube-api-access-jjfhm\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.199108 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.198988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-wtmp\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.199108 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.199046    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9215ec4-711e-400c-852c-aaaebbd7a7ec-metrics-client-ca\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.199174 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.199135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-accelerators-collector-config\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.300379 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300296    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-textfile\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.300379 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300349    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx6p7\" (UniqueName: \"kubernetes.io/projected/d9215ec4-711e-400c-852c-aaaebbd7a7ec-kube-api-access-hx6p7\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.300656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.300656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300410    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-root\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.300656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300438    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.300656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ff9623fb-ce84-42fc-aa52-fc263d66ec90-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.300656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.300656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300523    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.300656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.300656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300587    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.300656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmz9r\" (UniqueName: \"kubernetes.io/projected/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-api-access-jmz9r\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.301039 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.301039 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfhm\" (UniqueName: \"kubernetes.io/projected/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-kube-api-access-jjfhm\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.301039 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-wtmp\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.301039 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9215ec4-711e-400c-852c-aaaebbd7a7ec-metrics-client-ca\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.301039 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300796    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-accelerators-collector-config\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.301039 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9623fb-ce84-42fc-aa52-fc263d66ec90-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.301039 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-tls\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.301039 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300882    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-sys\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.301039 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.300967    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-sys\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.301481 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.301318    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-textfile\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.303498 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.302077    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-root\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.303498 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.302698    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-wtmp\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.303498 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.302910    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.303498 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.302981    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-accelerators-collector-config\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.303498 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.303321    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.303856 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.303541    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ff9623fb-ce84-42fc-aa52-fc263d66ec90-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.303856 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.303587    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9623fb-ce84-42fc-aa52-fc263d66ec90-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.304115 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.304051    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9215ec4-711e-400c-852c-aaaebbd7a7ec-metrics-client-ca\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.304512 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.304432    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:05.305922 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.305896    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.309873 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.309813    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.309873 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.309819    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.310906 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.310879    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.311256 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.311169    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.311434 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.311417    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-tls\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.327949 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.327920    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx6p7\" (UniqueName: \"kubernetes.io/projected/d9215ec4-711e-400c-852c-aaaebbd7a7ec-kube-api-access-hx6p7\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.328532 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.328506    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfhm\" (UniqueName: \"kubernetes.io/projected/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-kube-api-access-jjfhm\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.330549 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.330523    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmz9r\" (UniqueName: \"kubernetes.io/projected/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-api-access-jmz9r\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.363638 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.363600    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:05.426204 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.426171    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:05.430906 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.430880    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:05.466859 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.466831    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:05.558451 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.557091    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:05.697385 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.697329    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:05.701815 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.701734    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"]
Nov 02 12:27:05.704168 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:27:05.704143    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb243e5_8e99_4391_9a55_6e3a8f82ae0c.slice/crio-851cf5aa6fbc6747bee11fcc8c028d936ed2f893311ae001980a6a22cf92103b WatchSource:0}: Error finding container 851cf5aa6fbc6747bee11fcc8c028d936ed2f893311ae001980a6a22cf92103b: Status 404 returned error can't find the container with id 851cf5aa6fbc6747bee11fcc8c028d936ed2f893311ae001980a6a22cf92103b
Nov 02 12:27:05.714465 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.714436    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 12:27:05.714595 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.714577    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gr8bh\""
Nov 02 12:27:05.806389 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.806361    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"]
Nov 02 12:27:05.973239 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:27:05.973204    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9623fb_ce84_42fc_aa52_fc263d66ec90.slice/crio-db036f5ef9413c246a4091c304bee6732a34ae5a1d4a6c4352dfb0c18681e0a5 WatchSource:0}: Error finding container db036f5ef9413c246a4091c304bee6732a34ae5a1d4a6c4352dfb0c18681e0a5: Status 404 returned error can't find the container with id db036f5ef9413c246a4091c304bee6732a34ae5a1d4a6c4352dfb0c18681e0a5
Nov 02 12:27:05.999900 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:05.999819    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr" event={"ID":"ff9623fb-ce84-42fc-aa52-fc263d66ec90","Type":"ContainerStarted","Data":"db036f5ef9413c246a4091c304bee6732a34ae5a1d4a6c4352dfb0c18681e0a5"}
Nov 02 12:27:06.003623 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.002753    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lbm4n" event={"ID":"d9215ec4-711e-400c-852c-aaaebbd7a7ec","Type":"ContainerStarted","Data":"6d91a6143861ec9a16bae11c0ae9b49f872c5a9a0be4dbe6642c17e0123ec6ff"}
Nov 02 12:27:06.004819 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.004736    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx" event={"ID":"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c","Type":"ContainerStarted","Data":"3caaaa160138de69efc1ebb2ef76153fd043066fc3544d64920a39691b918abe"}
Nov 02 12:27:06.004819 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.004778    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx" event={"ID":"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c","Type":"ContainerStarted","Data":"851cf5aa6fbc6747bee11fcc8c028d936ed2f893311ae001980a6a22cf92103b"}
Nov 02 12:27:06.245334 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.245256    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:06.463199 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.463149    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:06.500525 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.498663    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 12:27:06.503716 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.501931    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.511646 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.511622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.511833 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.511819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.511982 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.511949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-web-config\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.512117 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.512073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.512117 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.512101    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.512237 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.512141    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.512237 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.512162    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.512237 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.512193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.512237 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.512218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.512416 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.512255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-out\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.512416 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.512296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw7dg\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-kube-api-access-lw7dg\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.512416 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.512323    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.512416 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.512352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.524021 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.523403    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 12:27:06.524021 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.523609    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 12:27:06.524021 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.523679    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 12:27:06.524021 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.523733    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 12:27:06.524773 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.524536    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 12:27:06.524773 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.524588    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 12:27:06.524773 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.524629    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 12:27:06.524773 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.524728    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-g5vwz\""
Nov 02 12:27:06.525132 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.525093    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 12:27:06.527135 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.527111    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 12:27:06.538168 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.538123    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 12:27:06.581424 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.581387    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.612739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw7dg\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-kube-api-access-lw7dg\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.612788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.612817    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.612852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.612880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.612911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-web-config\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.612971    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.613012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.613055    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.613079    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.613109    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.613140    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.614499 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.613165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-out\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.617624 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.615973    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.617624 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.617307    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.618736 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.618706    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.621732 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.621687    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.623621 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.623597    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.624868 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.624822    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.630505 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.630446    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.630631 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.630509    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.631010 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.630948    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.631194 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.631173    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-web-config\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.631540 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.631484    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-out\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.633725 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.633680    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.654135 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.654069    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw7dg\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-kube-api-access-lw7dg\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:06.831220 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:06.830906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:07.011231 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.010836    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx" event={"ID":"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c","Type":"ContainerStarted","Data":"11ae96cce6b9f81f4c3ca6d61325ce8c5d67bf5a790cec7db3e1dd44e7183dc2"}
Nov 02 12:27:07.022907 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.022835    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mqtc" event={"ID":"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a","Type":"ContainerStarted","Data":"99775d67d953cd9fc51fa30594037d582878f0180619cf28ca237f646595eae0"}
Nov 02 12:27:07.028332 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.027478    2582 generic.go:356] "Generic (PLEG): container finished" podID="d9215ec4-711e-400c-852c-aaaebbd7a7ec" containerID="0cf71450c60d2a04679a5649c48299e93443c80652c7232a92dfab707d501c4c" exitCode=0
Nov 02 12:27:07.028332 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.027526    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lbm4n" event={"ID":"d9215ec4-711e-400c-852c-aaaebbd7a7ec","Type":"ContainerDied","Data":"0cf71450c60d2a04679a5649c48299e93443c80652c7232a92dfab707d501c4c"}
Nov 02 12:27:07.186769 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.186735    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:07.190462 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.190318    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 12:27:07.197900 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:27:07.197858    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e92edc_76ab_4aac_9928_4fd21b20b7e8.slice/crio-b35a43d4116b66143b827d78db7d612bd73cfd3adfa90f575dab329f5fb440f6 WatchSource:0}: Error finding container b35a43d4116b66143b827d78db7d612bd73cfd3adfa90f575dab329f5fb440f6: Status 404 returned error can't find the container with id b35a43d4116b66143b827d78db7d612bd73cfd3adfa90f575dab329f5fb440f6
Nov 02 12:27:07.310957 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.310856    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:07.313932 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.313900    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57f985446f-hkswt"]
Nov 02 12:27:07.317026 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.316970    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.334021 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.333841    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 12:27:07.334021 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.333963    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 12:27:07.334642 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.334415    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 12:27:07.334642 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.334462    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 12:27:07.334642 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.334473    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 12:27:07.334642 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.334518    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5bwmg\""
Nov 02 12:27:07.334642 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.334591    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-67u7r1mgu2vji\""
Nov 02 12:27:07.348490 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.348452    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57f985446f-hkswt"]
Nov 02 12:27:07.421101 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.420694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-tls\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.421101 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.420733    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.421101 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.420779    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.421101 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.420796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.421101 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.420843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-metrics-client-ca\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.421101 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.420862    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76ddl\" (UniqueName: \"kubernetes.io/projected/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-kube-api-access-76ddl\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.421101 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.420891    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.421101 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.420918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-grpc-tls\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.421101 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.420946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-tmp\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.523301 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.521892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76ddl\" (UniqueName: \"kubernetes.io/projected/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-kube-api-access-76ddl\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.523301 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.521982    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.523301 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.522050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-grpc-tls\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.523301 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.522081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-tmp\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.523301 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.522115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-tls\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.523301 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.522150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.523301 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.522211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.523301 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.522240    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.523301 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.522282    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-metrics-client-ca\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.525490 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.524161    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-metrics-client-ca\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.525490 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.524637    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-tmp\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.529733 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.529680    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.530889 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.530337    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-tls\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.531502 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.531475    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.532184 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.532160    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-grpc-tls\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.533446 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.533424    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.536288 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.536261    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.549938 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.549910    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76ddl\" (UniqueName: \"kubernetes.io/projected/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-kube-api-access-76ddl\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.651615 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.651445    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:07.987581 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:07.986936    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57f985446f-hkswt"]
Nov 02 12:27:07.991829 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:27:07.991795    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc7e7ac_ba25_45f4_b6c9_5d7bff402d72.slice/crio-55b8e7e1aedc98e749e1eaba81690c1728619590490d37ce5ea814f260932c09 WatchSource:0}: Error finding container 55b8e7e1aedc98e749e1eaba81690c1728619590490d37ce5ea814f260932c09: Status 404 returned error can't find the container with id 55b8e7e1aedc98e749e1eaba81690c1728619590490d37ce5ea814f260932c09
Nov 02 12:27:08.035417 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.035368    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerStarted","Data":"b35a43d4116b66143b827d78db7d612bd73cfd3adfa90f575dab329f5fb440f6"}
Nov 02 12:27:08.038139 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.037931    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-snnjx" event={"ID":"9b72267e-e7f7-4115-af3e-14a382e659af","Type":"ContainerStarted","Data":"e8d65c37258b185cf39adfd1881ddc854d2507b223bf04cd4bec85142d127706"}
Nov 02 12:27:08.038139 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.037981    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-snnjx" event={"ID":"9b72267e-e7f7-4115-af3e-14a382e659af","Type":"ContainerStarted","Data":"e9dfb9d889e08633a128450cc98ac3cc5ceaf6ee36a5c3c455da190003545b40"}
Nov 02 12:27:08.038139 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.038063    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:08.041321 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.041290    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lbm4n" event={"ID":"d9215ec4-711e-400c-852c-aaaebbd7a7ec","Type":"ContainerStarted","Data":"7249bed1de0e6b8abbfbdbbf697644904c39eaba3e02417c5361b66f48cf144a"}
Nov 02 12:27:08.041485 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.041347    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lbm4n" event={"ID":"d9215ec4-711e-400c-852c-aaaebbd7a7ec","Type":"ContainerStarted","Data":"3e7d374e203d57f5441a507f838fa1393730078c2e15446d58d7213a145147cb"}
Nov 02 12:27:08.044402 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.044285    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt" event={"ID":"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72","Type":"ContainerStarted","Data":"55b8e7e1aedc98e749e1eaba81690c1728619590490d37ce5ea814f260932c09"}
Nov 02 12:27:08.047259 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.047231    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx" event={"ID":"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c","Type":"ContainerStarted","Data":"66b186ca23cae2da7d43e1ed33c4a2c8505959f73ffe774a7cfce6e89a2d5058"}
Nov 02 12:27:08.084509 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.083528    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-snnjx" podStartSLOduration=2.49592845 podStartE2EDuration="5.083507362s" totalImagesPullingTime="2.587578912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:08.082642853 +0000 UTC m=+45.919024835" watchObservedRunningTime="2026-11-02 12:27:08.083507362 +0000 UTC m=+45.919889247"
Nov 02 12:27:08.138400 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.137191    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx" podStartSLOduration=2.636932792 podStartE2EDuration="4.137146231s" totalImagesPullingTime="1.500213439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:08.135536897 +0000 UTC m=+45.971918801" watchObservedRunningTime="2026-11-02 12:27:08.137146231 +0000 UTC m=+45.973528113"
Nov 02 12:27:08.189181 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.189124    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lbm4n" podStartSLOduration=1.820820178 podStartE2EDuration="3.189110438s" totalImagesPullingTime="1.273978995s" totalInitContainerRuntime="94.311265ms" isStatefulPod=true podCreationTimestamp="2026-11-02 12:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:08.188888789 +0000 UTC m=+46.025270668" watchObservedRunningTime="2026-11-02 12:27:08.189110438 +0000 UTC m=+46.025492318"
Nov 02 12:27:08.298252 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.297187    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:08.427012 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.426952    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:08.551787 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:08.551671    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:09.060418 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.060166    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr" event={"ID":"ff9623fb-ce84-42fc-aa52-fc263d66ec90","Type":"ContainerStarted","Data":"5a97df308767c8e3a65b3e64aaa426dcfed0f60a49b9918c83b4274e25b5c04f"}
Nov 02 12:27:09.060868 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.060499    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr" event={"ID":"ff9623fb-ce84-42fc-aa52-fc263d66ec90","Type":"ContainerStarted","Data":"041ea1337cbc5c6285c9d89d07a612db50075075a4c5ca00faad0bac02d10d48"}
Nov 02 12:27:09.060868 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.060537    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr" event={"ID":"ff9623fb-ce84-42fc-aa52-fc263d66ec90","Type":"ContainerStarted","Data":"961ce29a3a242daf1159b9237c1dfdd7e84344a0db372d7bdac85cf5ae571a69"}
Nov 02 12:27:09.063246 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.063221    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mqtc" event={"ID":"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a","Type":"ContainerStarted","Data":"d96a4835acf63dc0e9edbf9267106dff3bd57a6dedd69e7cd8eee315dcd92f90"}
Nov 02 12:27:09.066627 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.066588    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nkq67" event={"ID":"4c8202fc-395e-4188-af53-19a2d40a55fe","Type":"ContainerStarted","Data":"1b366515e4709795a54c7f8288b3f895766eb2e257d0a073eed64bccd0a9f90a"}
Nov 02 12:27:09.068423 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.068398    2582 generic.go:356] "Generic (PLEG): container finished" podID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerID="8cc3c8ee3ebb3173b9b413353bd7104ed4d37655ff2e43588b51ebc2df7b728c" exitCode=0
Nov 02 12:27:09.069460 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.069419    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerDied","Data":"8cc3c8ee3ebb3173b9b413353bd7104ed4d37655ff2e43588b51ebc2df7b728c"}
Nov 02 12:27:09.101007 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.100960    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:09.107341 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.107284    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr" podStartSLOduration=2.027642118 podStartE2EDuration="4.1072671s" totalImagesPullingTime="2.079624982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:09.106160936 +0000 UTC m=+46.942542827" watchObservedRunningTime="2026-11-02 12:27:09.1072671 +0000 UTC m=+46.943648982"
Nov 02 12:27:09.150790 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.150726    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nkq67" podStartSLOduration=1.989372116 podStartE2EDuration="6.150709005s" totalImagesPullingTime="4.161336889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:09.1487606 +0000 UTC m=+46.985142483" watchObservedRunningTime="2026-11-02 12:27:09.150709005 +0000 UTC m=+46.987090887"
Nov 02 12:27:09.218320 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.218245    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:09.520015 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.519896    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8mqtc" podStartSLOduration=3.124585184 podStartE2EDuration="6.519870785s" totalImagesPullingTime="3.395285601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 12:27:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:09.252493897 +0000 UTC m=+47.088875777" watchObservedRunningTime="2026-11-02 12:27:09.519870785 +0000 UTC m=+47.356252677"
Nov 02 12:27:09.520590 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.520568    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f7db9d655-n8z5r"]
Nov 02 12:27:09.522959 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.522936    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.540252 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.540227    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 12:27:09.540376 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.540249    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 12:27:09.540376 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.540364    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 12:27:09.540572 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.540389    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 12:27:09.541070 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.541047    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-gnmlss15lorj\""
Nov 02 12:27:09.546514 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.544569    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f7db9d655-n8z5r"]
Nov 02 12:27:09.548526 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.548503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/14492824-39e2-4e98-a91d-ac81382ad70f-metrics-server-audit-profiles\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.548630 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.548552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-secret-metrics-server-client-certs\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.548695 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.548667    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-client-ca-bundle\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.548745 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.548704    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/14492824-39e2-4e98-a91d-ac81382ad70f-audit-log\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.548823 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.548788    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-secret-metrics-server-tls\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.548931 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.548844    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14492824-39e2-4e98-a91d-ac81382ad70f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.548931 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.548888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts8bt\" (UniqueName: \"kubernetes.io/projected/14492824-39e2-4e98-a91d-ac81382ad70f-kube-api-access-ts8bt\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.640877 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.640845    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-l6knz\""
Nov 02 12:27:09.649393 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.649360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/14492824-39e2-4e98-a91d-ac81382ad70f-metrics-server-audit-profiles\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.649540 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.649431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-secret-metrics-server-client-certs\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.649540 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.649477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-client-ca-bundle\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.649540 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.649506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/14492824-39e2-4e98-a91d-ac81382ad70f-audit-log\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.649695 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.649551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-secret-metrics-server-tls\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.649695 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.649578    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14492824-39e2-4e98-a91d-ac81382ad70f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.649695 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.649624    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts8bt\" (UniqueName: \"kubernetes.io/projected/14492824-39e2-4e98-a91d-ac81382ad70f-kube-api-access-ts8bt\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.650334 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.650302    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/14492824-39e2-4e98-a91d-ac81382ad70f-audit-log\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.650529 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.650504    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/14492824-39e2-4e98-a91d-ac81382ad70f-metrics-server-audit-profiles\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.650591 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.650575    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14492824-39e2-4e98-a91d-ac81382ad70f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.653287 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.653266    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-secret-metrics-server-tls\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.653401 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.653377    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-secret-metrics-server-client-certs\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.653718 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.653697    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-client-ca-bundle\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.682827 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.682787    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts8bt\" (UniqueName: \"kubernetes.io/projected/14492824-39e2-4e98-a91d-ac81382ad70f-kube-api-access-ts8bt\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.837309 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.837226    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:09.870371 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.870338    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"]
Nov 02 12:27:09.873547 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.873522    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:09.886709 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.886654    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:09.897224 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.896302    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 02 12:27:09.897224 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.896356    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-g8ktw\""
Nov 02 12:27:09.898440 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.898396    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"]
Nov 02 12:27:09.952013 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.951949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tkl\" (UniqueName: \"kubernetes.io/projected/c0067039-d47d-4ee4-8e02-1e24cdc26732-kube-api-access-b6tkl\") pod \"monitoring-plugin-77f4f76594-5d9mm\" (UID: \"c0067039-d47d-4ee4-8e02-1e24cdc26732\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:09.952195 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:09.952106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c0067039-d47d-4ee4-8e02-1e24cdc26732-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5d9mm\" (UID: \"c0067039-d47d-4ee4-8e02-1e24cdc26732\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:10.003568 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.003538    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:10.053224 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.053189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tkl\" (UniqueName: \"kubernetes.io/projected/c0067039-d47d-4ee4-8e02-1e24cdc26732-kube-api-access-b6tkl\") pod \"monitoring-plugin-77f4f76594-5d9mm\" (UID: \"c0067039-d47d-4ee4-8e02-1e24cdc26732\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:10.053363 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.053262    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c0067039-d47d-4ee4-8e02-1e24cdc26732-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5d9mm\" (UID: \"c0067039-d47d-4ee4-8e02-1e24cdc26732\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:10.056306 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.056281    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c0067039-d47d-4ee4-8e02-1e24cdc26732-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5d9mm\" (UID: \"c0067039-d47d-4ee4-8e02-1e24cdc26732\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:10.079058 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.079028    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tkl\" (UniqueName: \"kubernetes.io/projected/c0067039-d47d-4ee4-8e02-1e24cdc26732-kube-api-access-b6tkl\") pod \"monitoring-plugin-77f4f76594-5d9mm\" (UID: \"c0067039-d47d-4ee4-8e02-1e24cdc26732\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:10.151921 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.151897    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f7db9d655-n8z5r"]
Nov 02 12:27:10.185694 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.185657    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:10.371701 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.371664    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-cf7f64998-5wcsl"]
Nov 02 12:27:10.374420 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.374396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.391957 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.391370    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 12:27:10.391957 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.391370    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 12:27:10.391957 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.391456    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 12:27:10.391957 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.391456    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mbf6h\""
Nov 02 12:27:10.391957 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.391500    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 12:27:10.391957 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.391607    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 12:27:10.405872 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.405835    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 12:27:10.410301 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.410268    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cf7f64998-5wcsl"]
Nov 02 12:27:10.457003 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.456960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.457176 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.457013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.457176 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.457096    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-serving-certs-ca-bundle\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.457176 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.457114    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-metrics-client-ca\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.457176 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.457138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-telemeter-client-tls\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.457392 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.457221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-secret-telemeter-client\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.457392 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.457264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-federate-client-tls\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.457392 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.457304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4gb2\" (UniqueName: \"kubernetes.io/projected/025ece4b-9534-475f-b96d-0cb3edfe3ac3-kube-api-access-g4gb2\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.490410 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.490382    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"]
Nov 02 12:27:10.491008 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:27:10.490962    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0067039_d47d_4ee4_8e02_1e24cdc26732.slice/crio-30c18bb65aa5a2520d2f13157f87e6701b10f0ca336dff3e800114f451bf5fcf WatchSource:0}: Error finding container 30c18bb65aa5a2520d2f13157f87e6701b10f0ca336dff3e800114f451bf5fcf: Status 404 returned error can't find the container with id 30c18bb65aa5a2520d2f13157f87e6701b10f0ca336dff3e800114f451bf5fcf
Nov 02 12:27:10.558094 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.558061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4gb2\" (UniqueName: \"kubernetes.io/projected/025ece4b-9534-475f-b96d-0cb3edfe3ac3-kube-api-access-g4gb2\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.558238 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.558176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.558238 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.558206    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.558238 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.558231    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-serving-certs-ca-bundle\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.558477 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.558452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-metrics-client-ca\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.558540 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.558502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-telemeter-client-tls\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.558589 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.558548    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-secret-telemeter-client\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.559230 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.559207    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.559230 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.559217    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-metrics-client-ca\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.559396 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.559248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-federate-client-tls\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.559488 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.559467    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-serving-certs-ca-bundle\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.562086 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.561709    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.563177 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.563151    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-federate-client-tls\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.563302 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.563287    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-telemeter-client-tls\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.570465 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.570437    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-secret-telemeter-client\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.584683 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.584655    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4gb2\" (UniqueName: \"kubernetes.io/projected/025ece4b-9534-475f-b96d-0cb3edfe3ac3-kube-api-access-g4gb2\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:10.690774 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:10.690749    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:11.037684 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.037652    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-cf7f64998-5wcsl"]
Nov 02 12:27:11.080284 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.080245    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt" event={"ID":"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72","Type":"ContainerStarted","Data":"6a46bd5eab472740725f09f3fbafe887a8814b42ad3077194b18be0f1f7f6629"}
Nov 02 12:27:11.080748 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.080324    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt" event={"ID":"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72","Type":"ContainerStarted","Data":"81a9caa606226288449644337b3b2a24a213f58b2d8e3f3d79fb55bc62b96e03"}
Nov 02 12:27:11.080748 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.080364    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt" event={"ID":"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72","Type":"ContainerStarted","Data":"40c987c1960110ed5a91ac9709de51fe8629eb4193899f23cfb787b51aa50f22"}
Nov 02 12:27:11.081696 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.081671    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl" event={"ID":"025ece4b-9534-475f-b96d-0cb3edfe3ac3","Type":"ContainerStarted","Data":"8971abcc6165396c2ba45e7697f7e03d011c252b25c8d99ca87d518dce60cf33"}
Nov 02 12:27:11.083323 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.083298    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r" event={"ID":"14492824-39e2-4e98-a91d-ac81382ad70f","Type":"ContainerStarted","Data":"5dda63d8ced7535d751f9351f0ff65191cf54fa01ccad87dc71ecc3c4441493a"}
Nov 02 12:27:11.087113 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.087092    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerStarted","Data":"4e4bd02e09d614bad4b3122c985d25b55035e77442d8ec2bd69aee266edc7a9e"}
Nov 02 12:27:11.087200 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.087152    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerStarted","Data":"bb0a83fe82f634b603d022df5d15a9ac84487b419c85adfa4d471a95896c1767"}
Nov 02 12:27:11.087200 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.087194    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerStarted","Data":"71348250e126d8505c1fcee0d51e377d401696bdcfb4f9d9142ed67c75484ea0"}
Nov 02 12:27:11.087317 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.087230    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerStarted","Data":"bdb53a466aceccc533a9cde054805fccbcbf84f294acee9b78014508a5535fc4"}
Nov 02 12:27:11.087317 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.087274    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerStarted","Data":"dcadc026789767fa11102be451e08f929a46592db1cc93397533ff87277d072c"}
Nov 02 12:27:11.088626 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.088581    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm" event={"ID":"c0067039-d47d-4ee4-8e02-1e24cdc26732","Type":"ContainerStarted","Data":"30c18bb65aa5a2520d2f13157f87e6701b10f0ca336dff3e800114f451bf5fcf"}
Nov 02 12:27:11.542594 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.542564    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:11.692519 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.692430    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:11.965406 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.964230    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 12:27:11.970026 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.969977    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:11.991299 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.991272    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 12:27:11.991299 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.991295    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c8tgcn5qluvqu\""
Nov 02 12:27:11.991463 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.991409    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 12:27:11.991531 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.991519    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 12:27:11.991574 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.991558    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 12:27:11.991621 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.991610    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 12:27:11.991656 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.991640    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 12:27:11.991684 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.991674    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 12:27:11.991745 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.991728    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 12:27:11.991776 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.991757    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-v8788\""
Nov 02 12:27:11.991845 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.991821    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 12:27:11.991874 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.991850    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 12:27:11.992536 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:11.992511    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:12.005462 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.005436    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 12:27:12.005462 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.005436    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 12:27:12.006014 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.005970    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 12:27:12.006302 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.006280    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 12:27:12.010746 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.010726    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 12:27:12.074857 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.074827    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.074969 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.074875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075045 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.074976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075113 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075113 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075113 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075254 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075216    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075295 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075295 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075382 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075363    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075423 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075392    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4jzs\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-kube-api-access-j4jzs\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075488 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075533 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-config\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075533 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075627 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075627 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075612    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075710 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075747 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075747 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.075810 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.075754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.098030 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.097977    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerStarted","Data":"0073d1967d0b898124408fc1da498d48cba29fd20381c08d67433164d100f182"}
Nov 02 12:27:12.101573 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.101544    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt" event={"ID":"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72","Type":"ContainerStarted","Data":"d1445a813a9cb16d575f814534302d127bc95620fc29ec84436c9f5792f7dc1c"}
Nov 02 12:27:12.101672 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.101617    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt" event={"ID":"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72","Type":"ContainerStarted","Data":"a3e492d69be106e89214dc64986371e524cde21fdefe5c7b809e152a3d7f7b0c"}
Nov 02 12:27:12.101672 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.101658    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt" event={"ID":"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72","Type":"ContainerStarted","Data":"f3362130d25406683a62dea7681fe463a5413d133eb8715255073382008f58ef"}
Nov 02 12:27:12.103244 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.103220    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r" event={"ID":"14492824-39e2-4e98-a91d-ac81382ad70f","Type":"ContainerStarted","Data":"c3118365c74ae1dd0146df1399dc13301553d9f1a252af3d318cfa35e00f04ee"}
Nov 02 12:27:12.167548 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.167478    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.956193057 podStartE2EDuration="6.167458118s" totalImagesPullingTime="3.196821553s" totalInitContainerRuntime="14.443508ms" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:12.166064602 +0000 UTC m=+50.002446473" watchObservedRunningTime="2026-11-02 12:27:12.167458118 +0000 UTC m=+50.003840001"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.176787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.176828    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.176871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.176913    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.176955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177022    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177051    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177201    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177234    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177379    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jzs\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-kube-api-access-j4jzs\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177555    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-config\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.178503 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.179441 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.179441 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.177973    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.179441 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.178018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.179441 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.178087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.179441 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.178248    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.179441 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.178344    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.179441 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.178977    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.179441 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.179314    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.180972 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.180704    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.184659 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.184637    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.187180 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.187153    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.188829 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.188803    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.189344 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.189321    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.191881 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.191858    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.193568 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.193189    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.193568 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.193489    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.193568 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.193529    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.193754 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.193675    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.193968 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.193950    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.194064 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.193950    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.194564 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.194542    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.196146 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.196121    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-config\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.202959 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.202938    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jzs\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-kube-api-access-j4jzs\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.226563 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.226463    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt" podStartSLOduration=2.277494459 podStartE2EDuration="5.226447384s" totalImagesPullingTime="2.948952925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:12.225398355 +0000 UTC m=+50.061780236" watchObservedRunningTime="2026-11-02 12:27:12.226447384 +0000 UTC m=+50.062829263"
Nov 02 12:27:12.271520 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.271458    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r" podStartSLOduration=1.6672934270000002 podStartE2EDuration="3.271440194s" totalImagesPullingTime="1.604146767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:12.270982085 +0000 UTC m=+50.107363962" watchObservedRunningTime="2026-11-02 12:27:12.271440194 +0000 UTC m=+50.107822076"
Nov 02 12:27:12.317552 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.317524    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:12.643207 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:12.643175    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 12:27:12.650141 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:27:12.650115    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb2d74e_5ea9_48e3_98b8_53963e608786.slice/crio-e27ad6ea38d49de9df8fd5a312d081f44a213d984789d22dbc95b2788a87fb46 WatchSource:0}: Error finding container e27ad6ea38d49de9df8fd5a312d081f44a213d984789d22dbc95b2788a87fb46: Status 404 returned error can't find the container with id e27ad6ea38d49de9df8fd5a312d081f44a213d984789d22dbc95b2788a87fb46
Nov 02 12:27:13.109070 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:13.109035    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl" event={"ID":"025ece4b-9534-475f-b96d-0cb3edfe3ac3","Type":"ContainerStarted","Data":"3f48c41101b06da6c53a579bdbe084fdd78c2d0defa9ade97b5144c572f02c86"}
Nov 02 12:27:13.109699 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:13.109131    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl" event={"ID":"025ece4b-9534-475f-b96d-0cb3edfe3ac3","Type":"ContainerStarted","Data":"878d2c7b6204678c743da696145ca8e3ce32f94c743cd284359ce1b7e6cd7d20"}
Nov 02 12:27:13.109699 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:13.109155    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl" event={"ID":"025ece4b-9534-475f-b96d-0cb3edfe3ac3","Type":"ContainerStarted","Data":"261d0cadf5a69926279f44a0ac1ea579f0743d4bc6288a039b55f603c9783757"}
Nov 02 12:27:13.110741 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:13.110712    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm" event={"ID":"c0067039-d47d-4ee4-8e02-1e24cdc26732","Type":"ContainerStarted","Data":"6af0b56c559dbd9a889c6e2f652227ddb14c23631d49957d058d15185384118e"}
Nov 02 12:27:13.110922 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:13.110905    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:13.112294 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:13.112270    2582 generic.go:356] "Generic (PLEG): container finished" podID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerID="e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64" exitCode=0
Nov 02 12:27:13.112398 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:13.112376    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerDied","Data":"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"}
Nov 02 12:27:13.112475 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:13.112427    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerStarted","Data":"e27ad6ea38d49de9df8fd5a312d081f44a213d984789d22dbc95b2788a87fb46"}
Nov 02 12:27:13.113282 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:13.113121    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:13.116228 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:13.116210    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:13.159088 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:13.158960    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl" podStartSLOduration=1.404017248 podStartE2EDuration="3.158943684s" totalImagesPullingTime="1.754926436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:13.15853575 +0000 UTC m=+50.994917652" watchObservedRunningTime="2026-11-02 12:27:13.158943684 +0000 UTC m=+50.995325565"
Nov 02 12:27:13.259044 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:13.258970    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm" podStartSLOduration=2.7023766 podStartE2EDuration="4.258958415s" totalImagesPullingTime="1.556581815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:13.25811511 +0000 UTC m=+51.094496991" watchObservedRunningTime="2026-11-02 12:27:13.258958415 +0000 UTC m=+51.095340309"
Nov 02 12:27:16.128290 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:16.128259    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerStarted","Data":"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"}
Nov 02 12:27:17.099984 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:17.099949    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:17.135282 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:17.135248    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerStarted","Data":"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"}
Nov 02 12:27:17.135763 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:17.135324    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerStarted","Data":"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"}
Nov 02 12:27:17.135763 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:17.135374    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerStarted","Data":"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"}
Nov 02 12:27:17.135763 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:17.135412    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerStarted","Data":"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"}
Nov 02 12:27:17.135763 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:17.135454    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerStarted","Data":"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"}
Nov 02 12:27:17.214138 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:17.214085    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.313197133 podStartE2EDuration="6.214069768s" totalImagesPullingTime="2.883685463s" totalInitContainerRuntime="17.187172ms" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:17.212497569 +0000 UTC m=+55.048879450" watchObservedRunningTime="2026-11-02 12:27:17.214069768 +0000 UTC m=+55.050451654"
Nov 02 12:27:17.234518 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:17.234486    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:17.317981 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:17.317951    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:17.365270 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:17.365234    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:18.072848 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:18.072818    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:19.123908 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:19.123885    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:19.645061 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:19.645032    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:19.920567 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:19.920482    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:20.049241 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:20.049209    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:20.221795 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:20.221713    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:20.341456 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:20.341418    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:20.542505 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:20.542426    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:25.329219 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:25.329167    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:25.525281 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:25.525237    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:25.645181 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:25.645153    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:25.841486 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:25.841455    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:26.042781 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:26.042695    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:26.755024 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:26.754945    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:27.284305 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:27.284269    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:27.408420 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:27.408388    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:28.437099 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.437065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:28.437490 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.437115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:27:28.452340 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.452319    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 12:27:28.459653 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.459629    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:27:28.465780 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.465759    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 12:27:28.469981 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.469962    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:28.538376 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.538343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdjq\" (UniqueName: \"kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq\") pod \"network-check-target-2hzq7\" (UID: \"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60\") " pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:27:28.554698 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.554678    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 12:27:28.560759 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.560743    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gr8bh\""
Nov 02 12:27:28.565858 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.565832    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:28.574242 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.574221    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 12:27:28.581948 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.581924    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwdjq\" (UniqueName: \"kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq\") pod \"network-check-target-2hzq7\" (UID: \"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60\") " pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:27:28.755223 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.755147    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2lqbm\""
Nov 02 12:27:28.760292 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.760276    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:27:28.763226 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.763208    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l6cfl\""
Nov 02 12:27:28.768358 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.768342    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:27:28.865620 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:27:28.865579    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259d31ef_0ef7_4655_9121_88fd07468c38.slice/crio-b5dc2fd5be2e223841f09bb68b1aed9a57c34a8bbafeda69531422cb6a987424 WatchSource:0}: Error finding container b5dc2fd5be2e223841f09bb68b1aed9a57c34a8bbafeda69531422cb6a987424: Status 404 returned error can't find the container with id b5dc2fd5be2e223841f09bb68b1aed9a57c34a8bbafeda69531422cb6a987424
Nov 02 12:27:28.866196 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:28.866172    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x7jpb"]
Nov 02 12:27:29.055960 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:29.055937    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2vkgz"]
Nov 02 12:27:29.056593 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:27:29.056563    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78221f6f_2223_4ac2_92da_b763ff2a6b39.slice/crio-775f38a882bf57ebbe1940016f33cd09377cf171a3cd556ea991ccad62807b4f WatchSource:0}: Error finding container 775f38a882bf57ebbe1940016f33cd09377cf171a3cd556ea991ccad62807b4f: Status 404 returned error can't find the container with id 775f38a882bf57ebbe1940016f33cd09377cf171a3cd556ea991ccad62807b4f
Nov 02 12:27:29.080098 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:29.079972    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2hzq7"]
Nov 02 12:27:29.080888 ip-10-0-134-229 kubenswrapper[2582]: W1102 12:27:29.080867    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c61308_36a9_4eb4_9cc9_ac85bf1bdd60.slice/crio-4ec459b10e653210fec5cc58b0d25625802d678b9f00870815f6600ee2e24a03 WatchSource:0}: Error finding container 4ec459b10e653210fec5cc58b0d25625802d678b9f00870815f6600ee2e24a03: Status 404 returned error can't find the container with id 4ec459b10e653210fec5cc58b0d25625802d678b9f00870815f6600ee2e24a03
Nov 02 12:27:29.178413 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:29.178379    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x7jpb" event={"ID":"259d31ef-0ef7-4655-9121-88fd07468c38","Type":"ContainerStarted","Data":"b5dc2fd5be2e223841f09bb68b1aed9a57c34a8bbafeda69531422cb6a987424"}
Nov 02 12:27:29.179612 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:29.179587    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2hzq7" event={"ID":"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60","Type":"ContainerStarted","Data":"4ec459b10e653210fec5cc58b0d25625802d678b9f00870815f6600ee2e24a03"}
Nov 02 12:27:29.180722 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:29.180696    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vkgz" event={"ID":"78221f6f-2223-4ac2-92da-b763ff2a6b39","Type":"ContainerStarted","Data":"775f38a882bf57ebbe1940016f33cd09377cf171a3cd556ea991ccad62807b4f"}
Nov 02 12:27:29.839613 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:29.837631    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:29.839613 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:29.837705    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:31.194101 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:31.194059    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vkgz" event={"ID":"78221f6f-2223-4ac2-92da-b763ff2a6b39","Type":"ContainerStarted","Data":"fe9cb4ee3dfcc924952789cd62e5c766dc1a0b79243ab91aa9b1861ba6006dd1"}
Nov 02 12:27:31.194738 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:31.194702    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2vkgz" event={"ID":"78221f6f-2223-4ac2-92da-b763ff2a6b39","Type":"ContainerStarted","Data":"d736bb61cbb307b4a2e072c9516376ee6896310a69cd6d94f6392a432db145b8"}
Nov 02 12:27:31.239565 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:31.239483    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2vkgz" podStartSLOduration=67.193470707 podStartE2EDuration="1m8.239470043s" totalImagesPullingTime="1.045999336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:31.239161641 +0000 UTC m=+69.075543521" watchObservedRunningTime="2026-11-02 12:27:31.239470043 +0000 UTC m=+69.075851924"
Nov 02 12:27:32.508812 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:32.508774    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:32.661025 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:32.660922    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:33.205659 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:33.205627    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2hzq7" event={"ID":"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60","Type":"ContainerStarted","Data":"46574a5125be410cfb895736e124285ab30ae8d9caf48a956c8d84b8cbf52a84"}
Nov 02 12:27:33.205838 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:33.205742    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:27:33.249784 ip-10-0-134-229 kubenswrapper[2582]: I1102 12:27:33.249714    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2hzq7" podStartSLOduration=66.942107356 podStartE2EDuration="1m10.249691989s" totalImagesPullingTime="3.307584633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:33.248486542 +0000 UTC m=+71.084868423" watchObservedRunningTime="2026-11-02 12:27:33.249691989 +0000 UTC m=+71.086073873"
Nov 02 12:27:33.824805 ip-10-0-134-229 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 12:27:33.843470 ip-10-0-134-229 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 12:27:33.843648 ip-10-0-134-229 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 12:27:33.843839 ip-10-0-134-229 systemd[1]: kubelet.service: Consumed 4.871s CPU time.
Nov 02 12:27:33.858904 ip-10-0-134-229 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 12:27:33.926086 ip-10-0-134-229 kubenswrapper[5879]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:27:33.926086 ip-10-0-134-229 kubenswrapper[5879]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 12:27:33.926086 ip-10-0-134-229 kubenswrapper[5879]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:27:33.926086 ip-10-0-134-229 kubenswrapper[5879]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 12:27:33.928270 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928180    5879 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:27:33.928270 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928271    5879 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:27:33.928270 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928274    5879 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928280    5879 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928284    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928288    5879 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928291    5879 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928294    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928298    5879 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928301    5879 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928303    5879 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928306    5879 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928308    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928311    5879 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928314    5879 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928316    5879 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928319    5879 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928321    5879 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928324    5879 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928326    5879 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928328    5879 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:27:33.928370 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928331    5879 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928333    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928336    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928339    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928362    5879 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928365    5879 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928374    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928378    5879 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928382    5879 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928385    5879 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928388    5879 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928390    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928393    5879 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928396    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928399    5879 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928401    5879 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928404    5879 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928406    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928409    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:27:33.928831 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928411    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928414    5879 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928417    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928419    5879 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928422    5879 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928425    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928427    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928430    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928433    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928435    5879 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928438    5879 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928440    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928442    5879 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928445    5879 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928447    5879 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928450    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928453    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928456    5879 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928459    5879 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:27:33.929325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928462    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928464    5879 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928467    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928470    5879 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928472    5879 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928475    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928477    5879 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928481    5879 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928485    5879 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928488    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928490    5879 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928493    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928495    5879 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928498    5879 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928500    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928503    5879 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928505    5879 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928508    5879 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928510    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:27:33.929791 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928516    5879 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928519    5879 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928522    5879 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928525    5879 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928527    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928529    5879 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928532    5879 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928534    5879 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928537    5879 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928540    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928543    5879 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928546    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928548    5879 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928551    5879 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928553    5879 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928556    5879 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928558    5879 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928562    5879 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928564    5879 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928567    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:27:33.930318 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928569    5879 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928572    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928575    5879 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928577    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928580    5879 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928582    5879 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928585    5879 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.928587    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929438    5879 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929445    5879 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929448    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929451    5879 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929454    5879 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929460    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929463    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929466    5879 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929469    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929472    5879 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929474    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929477    5879 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:27:33.930815 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929479    5879 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929482    5879 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929485    5879 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929487    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929490    5879 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929493    5879 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929495    5879 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929498    5879 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929500    5879 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929503    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929505    5879 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929508    5879 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929510    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929513    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929516    5879 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929518    5879 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929522    5879 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929526    5879 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929530    5879 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:27:33.931325 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929532    5879 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929535    5879 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929537    5879 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929540    5879 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929542    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929545    5879 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929547    5879 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929550    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929553    5879 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929556    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929558    5879 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929560    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929563    5879 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929565    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929568    5879 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929571    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929573    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929575    5879 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929578    5879 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929581    5879 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:27:33.931794 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929584    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929587    5879 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929589    5879 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929591    5879 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929594    5879 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929596    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929599    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929601    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929603    5879 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929606    5879 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929608    5879 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929611    5879 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929613    5879 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929616    5879 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929618    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929621    5879 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929624    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929627    5879 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929629    5879 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:27:33.932312 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929632    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929635    5879 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929638    5879 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929641    5879 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929643    5879 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929646    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929649    5879 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929651    5879 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929654    5879 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929657    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929659    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929662    5879 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929664    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929666    5879 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929669    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929671    5879 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929674    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929676    5879 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929679    5879 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:27:33.932777 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929681    5879 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929683    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929686    5879 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929688    5879 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929691    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929693    5879 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929695    5879 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929698    5879 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929700    5879 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929706    5879 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929709    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929711    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929714    5879 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929719    5879 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929722    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929726    5879 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.929728    5879 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930258    5879 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930264    5879 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930267    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:27:33.933252 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930270    5879 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930273    5879 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930276    5879 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930279    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930281    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930284    5879 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930287    5879 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930289    5879 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930292    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930294    5879 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930297    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930299    5879 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930302    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930304    5879 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930307    5879 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930309    5879 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930312    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930314    5879 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930317    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930320    5879 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:27:33.933754 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930322    5879 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930328    5879 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930332    5879 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930335    5879 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930337    5879 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930340    5879 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930342    5879 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930345    5879 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930348    5879 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930351    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930353    5879 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930355    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930358    5879 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930361    5879 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930363    5879 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930366    5879 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930369    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930372    5879 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930374    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:27:33.934278 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930377    5879 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930379    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930382    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930384    5879 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930387    5879 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930390    5879 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930392    5879 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930395    5879 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930397    5879 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930399    5879 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930401    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930404    5879 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930407    5879 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930409    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930413    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930416    5879 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930419    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930424    5879 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930427    5879 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930431    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:27:33.934829 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930434    5879 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930438    5879 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930442    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930444    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930447    5879 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930449    5879 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930451    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930454    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930456    5879 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930459    5879 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930462    5879 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930464    5879 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930466    5879 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930469    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930471    5879 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930474    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930476    5879 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930478    5879 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930481    5879 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:27:33.935706 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930484    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930486    5879 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930489    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930491    5879 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930494    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930496    5879 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930499    5879 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930501    5879 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930506    5879 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930509    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930512    5879 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930514    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930517    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930519    5879 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930523    5879 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930528    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930532    5879 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930534    5879 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930537    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:27:33.936542 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930541    5879 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930544    5879 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930547    5879 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930549    5879 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930552    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.930555    5879 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930654    5879 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930666    5879 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930675    5879 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930682    5879 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930687    5879 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930691    5879 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930695    5879 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930700    5879 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930703    5879 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930706    5879 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930709    5879 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930712    5879 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930715    5879 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930718    5879 flags.go:64] FLAG: --cgroup-root=""
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930721    5879 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930727    5879 flags.go:64] FLAG: --client-ca-file=""
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930730    5879 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 12:27:33.937016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930732    5879 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930737    5879 flags.go:64] FLAG: --cluster-domain=""
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930740    5879 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930743    5879 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930747    5879 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930751    5879 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930755    5879 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930758    5879 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930761    5879 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930765    5879 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930768    5879 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930771    5879 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930774    5879 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930777    5879 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930780    5879 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930784    5879 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930787    5879 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930790    5879 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930793    5879 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930796    5879 flags.go:64] FLAG: --enable-server="true"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930799    5879 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930803    5879 flags.go:64] FLAG: --event-burst="100"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930807    5879 flags.go:64] FLAG: --event-qps="50"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930810    5879 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 12:27:33.937605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930813    5879 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930815    5879 flags.go:64] FLAG: --eviction-hard=""
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930821    5879 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930824    5879 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930827    5879 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930831    5879 flags.go:64] FLAG: --eviction-soft=""
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930835    5879 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930838    5879 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930841    5879 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930843    5879 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930846    5879 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930850    5879 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930853    5879 flags.go:64] FLAG: --feature-gates=""
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930857    5879 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930860    5879 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930864    5879 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930868    5879 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930871    5879 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930874    5879 flags.go:64] FLAG: --help="false"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930877    5879 flags.go:64] FLAG: --hostname-override="ip-10-0-134-229.ec2.internal"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930880    5879 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930883    5879 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930885    5879 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930889    5879 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 12:27:33.938232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930892    5879 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930895    5879 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930898    5879 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930900    5879 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930903    5879 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930906    5879 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930909    5879 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930912    5879 flags.go:64] FLAG: --kube-reserved=""
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930915    5879 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930918    5879 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930921    5879 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930923    5879 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930926    5879 flags.go:64] FLAG: --lock-file=""
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930930    5879 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930934    5879 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930937    5879 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930943    5879 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930946    5879 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930949    5879 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930951    5879 flags.go:64] FLAG: --logging-format="text"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930954    5879 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930958    5879 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930960    5879 flags.go:64] FLAG: --manifest-url=""
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930963    5879 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930968    5879 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 12:27:33.938816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930971    5879 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930975    5879 flags.go:64] FLAG: --max-pods="110"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930978    5879 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930981    5879 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.930984    5879 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931000    5879 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931004    5879 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931007    5879 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931010    5879 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931018    5879 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931021    5879 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931024    5879 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931027    5879 flags.go:64] FLAG: --pod-cidr=""
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931030    5879 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931032    5879 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931035    5879 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931038    5879 flags.go:64] FLAG: --port="10250"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931042    5879 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931045    5879 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0486a016b8c5edbbe"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931048    5879 flags.go:64] FLAG: --qos-reserved=""
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931052    5879 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931056    5879 flags.go:64] FLAG: --register-node="true"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931059    5879 flags.go:64] FLAG: --register-with-taints=""
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931063    5879 flags.go:64] FLAG: --registry-burst="10"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931066    5879 flags.go:64] FLAG: --registry-qps="5"
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931069    5879 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 12:27:33.939458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931072    5879 flags.go:64] FLAG: --reserved-memory=""
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931076    5879 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931079    5879 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931082    5879 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931085    5879 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931087    5879 flags.go:64] FLAG: --runonce="false"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931091    5879 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931094    5879 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931096    5879 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931100    5879 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931102    5879 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931106    5879 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931108    5879 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931111    5879 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931115    5879 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931117    5879 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931120    5879 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931123    5879 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931125    5879 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931128    5879 flags.go:64] FLAG: --system-cgroups=""
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931131    5879 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931137    5879 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931139    5879 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931143    5879 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931147    5879 flags.go:64] FLAG: --tls-min-version=""
Nov 02 12:27:33.940122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931150    5879 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931157    5879 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931161    5879 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931164    5879 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931167    5879 flags.go:64] FLAG: --v="2"
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931175    5879 flags.go:64] FLAG: --version="false"
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931179    5879 flags.go:64] FLAG: --vmodule=""
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931184    5879 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931187    5879 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931350    5879 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931354    5879 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931358    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931360    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931363    5879 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931366    5879 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931369    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931371    5879 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931374    5879 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931376    5879 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931379    5879 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931382    5879 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931385    5879 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:27:33.940816 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931387    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931390    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931392    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931395    5879 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931397    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931400    5879 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931402    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931405    5879 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931408    5879 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931410    5879 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931413    5879 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931417    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931421    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931424    5879 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931426    5879 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931429    5879 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931432    5879 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931434    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931437    5879 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931439    5879 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:27:33.941412 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931442    5879 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931444    5879 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931447    5879 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931449    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931452    5879 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931455    5879 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931458    5879 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931460    5879 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931462    5879 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931465    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931467    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931469    5879 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931472    5879 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931475    5879 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931477    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931479    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931482    5879 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931484    5879 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931487    5879 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:27:33.941909 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931490    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931492    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931495    5879 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931497    5879 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931501    5879 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931505    5879 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931507    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931510    5879 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931512    5879 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931517    5879 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931520    5879 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931523    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931526    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931529    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931531    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931534    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931537    5879 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931539    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931541    5879 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931544    5879 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:27:33.942386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931547    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931550    5879 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931552    5879 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931554    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931557    5879 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931560    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931562    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931565    5879 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931567    5879 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931569    5879 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931572    5879 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931574    5879 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931577    5879 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931580    5879 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931582    5879 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931585    5879 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931588    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931592    5879 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931596    5879 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:27:33.942917 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931599    5879 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931602    5879 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931605    5879 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931608    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931610    5879 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931613    5879 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931615    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931618    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931621    5879 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931623    5879 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931626    5879 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931628    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931630    5879 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931633    5879 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931636    5879 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.931661    5879 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:27:33.943468 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931698    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931703    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931706    5879 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931709    5879 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931712    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931715    5879 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931718    5879 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931721    5879 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931724    5879 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931727    5879 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931730    5879 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931732    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931735    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931740    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931742    5879 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931747    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931750    5879 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931752    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931755    5879 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931758    5879 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:27:33.943910 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931761    5879 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931763    5879 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931766    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931768    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931782    5879 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931786    5879 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931789    5879 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931792    5879 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931796    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931799    5879 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931802    5879 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931805    5879 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931809    5879 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931812    5879 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931814    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931817    5879 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931819    5879 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931821    5879 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931824    5879 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931827    5879 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:27:33.944460 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931829    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931832    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931835    5879 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931837    5879 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931839    5879 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931843    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931846    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931849    5879 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931851    5879 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931854    5879 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931856    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931859    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931861    5879 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931864    5879 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931866    5879 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931869    5879 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931871    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931874    5879 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931876    5879 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931880    5879 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:27:33.944964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931884    5879 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931886    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931889    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931892    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931894    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931897    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931899    5879 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931902    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931904    5879 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931906    5879 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931909    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931912    5879 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931914    5879 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931917    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931919    5879 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931921    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931924    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931928    5879 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931930    5879 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931933    5879 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:27:33.945796 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931935    5879 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931938    5879 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931940    5879 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931943    5879 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931945    5879 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931947    5879 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931950    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931952    5879 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931956    5879 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931959    5879 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931962    5879 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931965    5879 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931968    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931971    5879 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931973    5879 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931975    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931978    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931982    5879 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931984    5879 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:27:33.946428 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.931986    5879 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.932001    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.932004    5879 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.932007    5879 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.932009    5879 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.932012    5879 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.932015    5879 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.937559    5879 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.937576    5879 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.937591    5879 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.937596    5879 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937659    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937665    5879 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937670    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937673    5879 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937676    5879 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937679    5879 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937682    5879 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937684    5879 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:27:33.946905 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937687    5879 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937690    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937693    5879 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937696    5879 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937698    5879 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937701    5879 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937703    5879 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937706    5879 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937709    5879 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937711    5879 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937715    5879 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937719    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937722    5879 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937724    5879 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937727    5879 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937730    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937733    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937735    5879 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937738    5879 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:27:33.947396 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937741    5879 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937743    5879 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937746    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937748    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937751    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937753    5879 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937756    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937758    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937761    5879 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937764    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937766    5879 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937769    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937771    5879 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937774    5879 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937776    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937779    5879 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937782    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937785    5879 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937788    5879 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937790    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:27:33.947872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937792    5879 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937795    5879 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937798    5879 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937800    5879 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937803    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937805    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937808    5879 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937811    5879 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937814    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937817    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937820    5879 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937822    5879 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937825    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937828    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937831    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937834    5879 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937836    5879 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937839    5879 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937841    5879 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937844    5879 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:27:33.948367 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937847    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937849    5879 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937851    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937854    5879 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937856    5879 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937859    5879 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937861    5879 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937864    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937866    5879 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937869    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937871    5879 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937874    5879 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937877    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937880    5879 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937883    5879 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937887    5879 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937889    5879 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937892    5879 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937895    5879 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:27:33.949046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937897    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937900    5879 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937903    5879 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937905    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937908    5879 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937911    5879 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937913    5879 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937916    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937918    5879 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937921    5879 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937924    5879 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937926    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937929    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937931    5879 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937934    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937936    5879 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937938    5879 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937941    5879 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937944    5879 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:27:33.949527 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.937946    5879 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.937975    5879 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938185    5879 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938191    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938194    5879 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938197    5879 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938200    5879 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938203    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938206    5879 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938208    5879 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938211    5879 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938214    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938216    5879 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938218    5879 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938221    5879 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938224    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:27:33.950088 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938226    5879 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938229    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938231    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938234    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938237    5879 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938240    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938242    5879 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938245    5879 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938248    5879 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938251    5879 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938254    5879 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938256    5879 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938258    5879 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938262    5879 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938266    5879 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938269    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938271    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938274    5879 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938276    5879 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:27:33.950667 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938279    5879 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938282    5879 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938284    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938287    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938289    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938292    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938295    5879 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938297    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938299    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938302    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938305    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938307    5879 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938310    5879 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938312    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938315    5879 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938318    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938321    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938323    5879 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938326    5879 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938328    5879 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:27:33.951332 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938331    5879 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938333    5879 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938336    5879 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938338    5879 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938341    5879 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938343    5879 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938346    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938348    5879 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938351    5879 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938355    5879 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938358    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938362    5879 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938365    5879 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938368    5879 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938371    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938373    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938376    5879 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938378    5879 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938382    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:27:33.951822 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938384    5879 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938387    5879 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938389    5879 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938392    5879 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938394    5879 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938397    5879 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938399    5879 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938402    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938405    5879 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938408    5879 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938410    5879 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938413    5879 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938415    5879 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938418    5879 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938420    5879 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938423    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938425    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938428    5879 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938430    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938433    5879 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:27:33.952347 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938435    5879 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938438    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938440    5879 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938443    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938445    5879 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938450    5879 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938453    5879 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938455    5879 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938457    5879 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938460    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938463    5879 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938466    5879 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938468    5879 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938471    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.938495    5879 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938529    5879 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 12:27:33.952827 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938534    5879 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938537    5879 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938540    5879 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938543    5879 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938545    5879 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938548    5879 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938553    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938556    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938559    5879 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938561    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938564    5879 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938566    5879 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938569    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938572    5879 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938574    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938577    5879 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938579    5879 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938582    5879 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938584    5879 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938587    5879 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 12:27:33.953438 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938589    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938593    5879 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938596    5879 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938599    5879 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938601    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938604    5879 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938607    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938609    5879 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938612    5879 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938614    5879 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938617    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938620    5879 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938622    5879 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938624    5879 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938627    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938629    5879 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938632    5879 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938634    5879 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938638    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 12:27:33.953928 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938641    5879 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938644    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938646    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938649    5879 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938651    5879 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938654    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938656    5879 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938659    5879 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938661    5879 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938663    5879 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938666    5879 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938668    5879 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938671    5879 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938674    5879 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938679    5879 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938682    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938685    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938687    5879 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938690    5879 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 12:27:33.954449 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938692    5879 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938695    5879 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938698    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938700    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938703    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938705    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938708    5879 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938710    5879 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938713    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938715    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938718    5879 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938720    5879 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938724    5879 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938727    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938730    5879 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938732    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938735    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938737    5879 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938740    5879 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938742    5879 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 12:27:33.954920 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938745    5879 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938747    5879 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938750    5879 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938752    5879 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938755    5879 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938758    5879 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938760    5879 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938764    5879 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938767    5879 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938771    5879 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938775    5879 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938778    5879 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938780    5879 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938783    5879 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938785    5879 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938788    5879 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938790    5879 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938793    5879 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938796    5879 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 12:27:33.955432 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938798    5879 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938801    5879 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938803    5879 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938806    5879 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938808    5879 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938812    5879 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938815    5879 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:33.938818    5879 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.938870    5879 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.939844    5879 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.939912    5879 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.940119    5879 server.go:1014] "Starting client certificate rotation"
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.940265    5879 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.940321    5879 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 12:21:23 +0000 UTC" deadline="2028-07-29 09:59:17.498985158 +0000 UTC"
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.940357    5879 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15237h31m43.558630602s"
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.940561    5879 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:27:33.955935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.941739    5879 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 12:27:33.956405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.943444    5879 log.go:25] "Validated CRI v1 runtime API"
Nov 02 12:27:33.956405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.945476    5879 log.go:25] "Validated CRI v1 image API"
Nov 02 12:27:33.956405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.946408    5879 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 12:27:33.956405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.949070    5879 fs.go:135] Filesystem UUIDs: map[64a92448-e7b0-477f-8772-79b4d7158201:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 feb98b30-4d11-401b-be6b-4da7db1e3340:/dev/nvme0n1p3]
Nov 02 12:27:33.956531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.949090    5879 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/157a29163cc443f325ee6ca2d4bb24477fe7d5c24fad4cb0a6bdbaefb07c52f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/157a29163cc443f325ee6ca2d4bb24477fe7d5c24fad4cb0a6bdbaefb07c52f7/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/30c18bb65aa5a2520d2f13157f87e6701b10f0ca336dff3e800114f451bf5fcf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/30c18bb65aa5a2520d2f13157f87e6701b10f0ca336dff3e800114f451bf5fcf/userdata/shm major:0 minor:361 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ec459b10e653210fec5cc58b0d25625802d678b9f00870815f6600ee2e24a03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ec459b10e653210fec5cc58b0d25625802d678b9f00870815f6600ee2e24a03/userdata/shm major:0 minor:505 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55b8e7e1aedc98e749e1eaba81690c1728619590490d37ce5ea814f260932c09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55b8e7e1aedc98e749e1eaba81690c1728619590490d37ce5ea814f260932c09/userdata/shm major:0 minor:308 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5927fee2edddf17965fe3e93cea6caa0de3abc9a67344b2da5770977d8c334d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5927fee2edddf17965fe3e93cea6caa0de3abc9a67344b2da5770977d8c334d7/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5dda63d8ced7535d751f9351f0ff65191cf54fa01ccad87dc71ecc3c4441493a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5dda63d8ced7535d751f9351f0ff65191cf54fa01ccad87dc71ecc3c4441493a/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69c6d5dffcb708513d462969fcde74577a883a833d0af76bbadf3a54581c7d81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69c6d5dffcb708513d462969fcde74577a883a833d0af76bbadf3a54581c7d81/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d91a6143861ec9a16bae11c0ae9b49f872c5a9a0be4dbe6642c17e0123ec6ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d91a6143861ec9a16bae11c0ae9b49f872c5a9a0be4dbe6642c17e0123ec6ff/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7384673b07504ae5d5827772219deea4aa974c328d8347c76298f5bb4884961d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7384673b07504ae5d5827772219deea4aa974c328d8347c76298f5bb4884961d/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/775f38a882bf57ebbe1940016f33cd09377cf171a3cd556ea991ccad62807b4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/775f38a882bf57ebbe1940016f33cd09377cf171a3cd556ea991ccad62807b4f/userdata/shm major:0 minor:503 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/851cf5aa6fbc6747bee11fcc8c028d936ed2f893311ae001980a6a22cf92103b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/851cf5aa6fbc6747bee11fcc8c028d936ed2f893311ae001980a6a22cf92103b/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8971abcc6165396c2ba45e7697f7e03d011c252b25c8d99ca87d518dce60cf33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8971abcc6165396c2ba45e7697f7e03d011c252b25c8d99ca87d518dce60cf33/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b35a43d4116b66143b827d78db7d612bd73cfd3adfa90f575dab329f5fb440f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b35a43d4116b66143b827d78db7d612bd73cfd3adfa90f575dab329f5fb440f6/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5dc2fd5be2e223841f09bb68b1aed9a57c34a8bbafeda69531422cb6a987424/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5dc2fd5be2e223841f09bb68b1aed9a57c34a8bbafeda69531422cb6a987424/userdata/shm major:0 minor:500 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7b8bfaaab6e0d96223a2d994677ddcd08d7d624ffa75453543f3dc158adaf4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7b8bfaaab6e0d96223a2d994677ddcd08d7d624ffa75453543f3dc158adaf4e/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8c1a1eb2cfe560549d2d35c957c0c252df53c896f45ae08a1e00315e363a389/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8c1a1eb2cfe560549d2d35c957c0c252df53c896f45ae08a1e00315e363a389/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba748218a38311cf4bf3a380834741cab72cab6a0a9c885f0a9a23cc6241f87c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba748218a38311cf4bf3a380834741cab72cab6a0a9c885f0a9a23cc6241f87c/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca3d73812b549dbba9eec19c804667161aa5f128dfb36b0096e84230fe72d876/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca3d73812b549dbba9eec19c804667161aa5f128dfb36b0096e84230fe72d876/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf82277f49d8e3ca224e9eb33c605be11e0445d211276acc34b917b79b268979/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf82277f49d8e3ca224e9eb33c605be11e0445d211276acc34b917b79b268979/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d064abf07fa1a44346d0f42e8579fc1e71d70a91f06f29e04286221680a236d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d064abf07fa1a44346d0f42e8579fc1e71d70a91f06f29e04286221680a236d3/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d7ceb6da9aa7950150360aef73ffba03976248108c7724d3f6320157a3b2a531/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d7ceb6da9aa7950150360aef73ffba03976248108c7724d3f6320157a3b2a531/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db036f5ef9413c246a4091c304bee6732a34ae5a1d4a6c4352dfb0c18681e0a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db036f5ef9413c246a4091c304bee6732a34ae5a1d4a6c4352dfb0c18681e0a5/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de9c9447ce3611f1658dd55296dfade31452a41ea352a1d821aafda26255bb78/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de9c9447ce3611f1658dd55296dfade31452a41ea352a1d821aafda26255bb78/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e27ad6ea38d49de9df8fd5a312d081f44a213d984789d22dbc95b2788a87fb46/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e27ad6ea38d49de9df8fd5a312d081f44a213d984789d22dbc95b2788a87fb46/userdata/shm major:0 minor:447 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f83efbb007fca102cf724127bf3e9d727ccf747a86eddefcd6ff5e20c45adaad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f83efbb007fca102cf724127bf3e9d727ccf747a86eddefcd6ff5e20c45adaad/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0162c6de-30e1-4e0b-949e-43b813b1e953/volumes/kubernetes.io~projected/kube-api-access-pv5cq:{mountpoint:/var/lib/kubelet/pods/0162c6de-30e1-4e0b-949e-43b813b1e953/volumes/kubernetes.io~projected/kube-api-access-pv5cq major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~projected/kube-api-access-g4gb2:{mountpoint:/var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~projected/kube-api-access-g4gb2 major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cf6192d-d2e9-498b-b582-32305e1572d7/volumes/kubernetes.io~projected/kube-api-access-qh9vf:{mountpoint:/var/lib/kubelet/pods/0cf6192d-d2e9-498b-b582-32305e1572d7/volumes/kubernetes.io~projected/kube-api-access-qh9vf major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14492824-39e2-4e98-a91d-ac81382ad70f/volumes/kubernetes.io~projected/kube-api-access-ts8bt:{mountpoint:/var/lib/kubelet/pods/14492824-39e2-4e98-a91d-ac81382ad70f/volumes/kubernetes.io~projected/kube-api-access-ts8bt major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14492824-39e2-4e98-a91d-ac81382ad70f/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/14492824-39e2-4e98-a91d-ac81382ad70f/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14492824-39e2-4e98-a91d-ac81382ad70f/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/14492824-39e2-4e98-a91d-ac81382ad70f/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14492824-39e2-4e98-a91d-ac81382ad70f/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/14492824-39e2-4e98-a91d-ac81382ad70f/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1827f1a3-a7e6-4d57-907a-763220ce3e0e/volumes/kubernetes.io~projected/kube-api-access-gxbqz:{mountpoint:/var/lib/kubelet/pods/1827f1a3-a7e6-4d57-907a-763220ce3e0e/volumes/kubernetes.io~projected/kube-api-access-gxbqz major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60/volumes/kubernetes.io~projected/kube-api-access-fwdjq:{mountpoint:/var/lib/kubelet/pods/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60/volumes/kubernetes.io~projected/kube-api-access-fwdjq major:0 minor:502 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/259d31ef-0ef7-4655-9121-88fd07468c38/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/259d31ef-0ef7-4655-9121-88fd07468c38/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:499 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~projected/kube-api-access-76ddl:{mountpoint:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~projected/kube-api-access-76ddl major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c8202fc-395e-4188-af53-19a2d40a55fe/volumes/kubernetes.io~projected/kube-api-access-5p26c:{mountpoint:/var/lib/kubelet/pods/4c8202fc-395e-4188-af53-19a2d40a55fe/volumes/kubernetes.io~projected/kube-api-access-5p26c major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c8202fc-395e-4188-af53-19a2d40a55fe/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4c8202fc-395e-4188-af53-19a2d40a55fe/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c/volumes/kubernetes.io~projected/kube-api-access-jjfhm:{mountpoint:/var/lib/kubelet/pods/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c/volumes/kubernetes.io~projected/kube-api-access-jjfhm major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471/volumes/kubernetes.io~projected/kube-api-access-97wkz:{mountpoint:/var/lib/kubelet/pods/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471/volumes/kubernetes.io~projected/kube-api-access-97wkz major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volume-subpaths/web-config/alertmanager/9 major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volume-subpaths/web-config/config-reloader/6 major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~empty-dir/config-out major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~projected/kube-api-access-lw7dg:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~projected/kube-api-access-lw7dg major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~projected/tls-assets major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/config-volume major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/web-config major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/718bc5cd-de70-4920-9e8f-5bc14fe1f45c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/718bc5cd-de70-4920-9e8f-5bc14fe1f45c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/718bc5cd-de70-4920-9e8f-5bc14fe1f45c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/718bc5cd-de70-4920-9e8f-5bc14fe1f45c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/718bc5cd-de70-4920-9e8f-5bc14fe1f45c/volumes/kubernetes.io~projected/kube-api-access-p8bmf:{mountpoint:/var/lib/kubelet/pods/718bc5cd-de70-4920-9e8f-5bc14fe1f45c/volumes/kubernetes.io~projected/kube-api-access-p8bmf major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78221f6f-2223-4ac2-92da-b763ff2a6b39/volumes/kubernetes.io~projected/kube-api-access-p8dqb:{mountpoint:/var/lib/kubelet/pods/78221f6f-2223-4ac2-92da-b763ff2a6b39/volumes/kubernetes.io~projected/kube-api-access-p8dqb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78221f6f-2223-4ac2-92da-b763ff2a6b39/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/78221f6f-2223-4ac2-92da-b763ff2a6b39/volumes/kubernetes.io~secret/metrics-certs major:0 minor:495 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a/volumes/kubernetes.io~projected/kube-api-access-xnwn9:{mountpoint:/var/lib/kubelet/pods/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a/volumes/kubernetes.io~projected/kube-api-access-xnwn9 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b72267e-e7f7-4115-af3e-14a382e659af/volumes/kubernetes.io~projected/kube-api-access-csd2s:{mountpoint:/var/lib/kubelet/pods/9b72267e-e7f7-4115-af3e-14a382e659af/volumes/kubernetes.io~projected/kube-api-access-csd2s major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b72267e-e7f7-4115-af3e-14a382e659af/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/9b72267e-e7f7-4115-af3e-14a382e659af/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a41b8e21-8e18-4e58-85d9-c24ba888b2f6/volumes/kubernetes.io~projected/kube-api-access-c5x28:{mountpoint:/var/lib/kubelet/pods/a41b8e21-8e18-4e58-85d9-c24ba888b2f6/volumes/kubernetes.io~projected/kube-api-access-c5x28 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b89c2be7-3500-4d9a-ac82-4ef367b83e3e/volumes/kubernetes.io~projected/kube-api-access-wvmb9:{mountpoint:/var/lib/kubelet/pods/b89c2be7-3500-4d9a-ac82-4ef367b83e3e/volumes/kubernetes.io~projected/kube-api-access-wvmb9 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0067039-d47d-4ee4-8e02-1e24cdc26732/volumes/kubernetes.io~projected/kube-api-access-b6tkl:{mountpoint:/var/lib/kubelet/pods/c0067039-d47d-4ee4-8e02-1e24cdc26732/volumes/kubernetes.io~projected/kube-api-access-b6tkl major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0067039-d47d-4ee4-8e02-1e24cdc26732/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/c0067039-d47d-4ee4-8e02-1e24cdc26732/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d59c905e-0042-4777-a2fe-d5d5ae04e47d/volumes/kubernetes.io~projected/kube-api-access-vgpqc:{mountpoint:/var/lib/kubelet/pods/d59c905e-0042-4777-a2fe-d5d5ae04e47d/volumes/kubernetes.io~projected/kube-api-access-vgpqc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9215ec4-711e-400c-852c-aaaebbd7a7ec/volumes/kubernetes.io~projected/kube-api-access-hx6p7:{mountpoint:/var/lib/kubelet/pods/d9215ec4-711e-400c-852c-aaaebbd7a7ec/volumes/kubernetes.io~projected/kube-api-access-hx6p7 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9215ec4-711e-400c-852c-aaaebbd7a7ec/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d9215ec4-711e-400c-852c-aaaebbd7a7ec/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d9215ec4-711e-400c-852c-aaaebbd7a7ec/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/d9215ec4-711e-400c-852c-aaaebbd7a7ec/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff9623fb-ce84-42fc-aa52-fc263d66ec90/volumes/kubernetes.io~projected/kube-api-access-jmz9r:{mountpoint:/var/lib/kubelet/pods/ff9623fb-ce84-42fc-aa52-fc263d66ec90/volumes/kubernetes.io~projected/kube-api-access-jmz9r major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff9623fb-ce84-42fc-aa52-fc263d66ec90/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ff9623fb-ce84-42fc-aa52-fc263d66ec90/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff9623fb-ce84-42fc-aa52-fc263d66ec90/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ff9623fb-ce84-42fc-aa52-fc263d66ec90/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volume-subpaths/web-config/config-reloader/2 major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volume-subpaths/web-config/prometheus/15 major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~empty-dir/config-out major:0 minor:438 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~projected/kube-api-access-j4jzs:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~projected/kube-api-access-j4jzs major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~projected/tls-assets major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/config major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:442 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:439 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/web-config major:0 minor:441 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/523f7cf54286c60ffc87a525acdce2e0fa198c30a8bd02dc00fd1a2ebf8457cc/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/86e0b89bc75165c33e110eb992ff2373d9e77370dcb168a50b20e422c3e3a575/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5a85f2b27000b3ac11644ab2376b3324470576f8c7711a720dffb8706769fb9b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/b93ba4d1daafaf61b037965d28103e08b7a3c96f3877a1d20990d23e0808a53d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c1fb7687526b047c8bff4e0ee540a2bacaa5b5c042ba8178882e29341adb38eb/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/168bc6337841b6ff79c849d11ffa2f7102c18ee2d6df4b378db5b3a991a343ad/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/629a075ed6cf6718bfee488a0ca24c16a636cd3556932f44f8e64d7eea45f2d4/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/83e78a38f4cd8266c0cb62f04641367d12a13937d12c1e961e9fb89f6d563efb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d0350a5577d12c2c386dab7dfabf1fa32fbb22499df0f3d663014ee4efc471ac/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/24e365cde1568fce34f4e68942b461ea1ee829963a3d11c76aa883cb2d0a34d6/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/c1c8c9bf0a76e4f31c8287d7b34a4e631c4c5f52c91531f436aca314fb4f6a72/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/0960d0960072e04964d32adbe7c630e2ffa15ba1f463704cdae2af127244958e/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/805797e3db99284690678c473597b5d47f045f59a377f613a0a5a29724b59408/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/2a296453c01cd5496e4d31f9248c5c6308e8c3e85d9d3491bf4c3ce6a61af4bb/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e775647f8f0432eb13db04a3046c964c369b140e7fec7d54a06b62047e8234dc/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9d5bab3916c540cccf149d1c5736c91811080bc025e1a92a9827a3b548e7acd9/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/162416e1ea81c52ff0cc484d33ecfefcd5cf0d99f71bcdda5844e1391854066e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d06e4ac76120012012854f6628130186aac071d0d2244221f50a8948dfdc4bc2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/fd28084e866bd4575194cce428d5982856f7f410f7c675bfd9f6b482979c372b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/447fa13a200710501dfa88d3a9377c0c2e4f05973390ffe2affe48a953890d4d/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b2a3fc21c912b4ac9858c6570d30dfa4ac105ddd0453186fd9fc6d5eb67a429f/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/c38ff7f35f5074e4a28592d25d71038a277123f1cef65b3083d89bd63fc7024e/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/214453601a839ef0f69994f2b99d2b128f9bcb6297a282f7afe69572677037f8/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/e007271a4893ea96cdeeb678c2a1bb86ce75fcf82055691cdfc2d0de96aea755/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/2914ae1f7e0c7c4281e2a4647a519e0985cb2dabfc849a3f128dc23789217644/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/49c0555e025bad14a766eddc96abce677417a9d473f8987dc22fc9813aa692c7/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/bc182a46139e2ca35f35cd4ba915e97f31c442aeed0de4234c0316c81baddc40/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/75a6e33299cd5bcd1a5c6898fa7b0aff26cdf6ba59422b8f620a3896415b7070/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/4a576ff89fe37a28c01ad5305da5ea3de1e5136281cca41494c2fc8a8b689cbf/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/5a586eb0b35ce6e7cb263feac9a40ff4e9f91b6e603c28bb2150c83b6a049f57/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/1eb0c4319ff6aecf1e62fb5091d97852b66c8c7f72b39ceef93b992519e69155/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/81e7ba81b1fc4073625f029b511c9408fa9f86adf592b08ea4209201427e3d1f/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/669ca4010129e96ba622a67e71c9dac127e3e9f71bf138d79c76d87cb9733abb/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/bc1a1e6dadff1e2e058746e506ffd220b67b099bda11d436570ebacf616e88f4/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/552ae7f36a59e4bf01f84970347ed9c05ba2f89be40b8f246d90237d31e24f7a/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/d610490bb84a56b262fab45017e32d3c2ce8240fdf24faf93b2eac7835964043/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/5c3f4b010d1124aad8982a3925960bcc17fb4463d984540413ae1b445140efd6/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/a0cf19d45c448851a7c106e45cdd5fa73869f68259d61b95e757b8ecb156a978/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/df3558975498dbd79037d22dc07ebd394bb555d5ae976126c535d0d895da547e/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/7f3184dc1d1389a03b8cc1cf3f52e8c5a4fb4d54ceca9d0c4d60333bba74a451/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/4ccaeb91170d48333d713fb3321757b3e78197d923698b77d597399ea0ba6410/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/a4853fffb99a235fc1cd028c8c54d1e9dcba2ea4ac42b5351d94222ce5130964/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/ba19348b8df188016d81f3dd9e71a281b3f1d77cb3ca9cee336959d194deebea/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/14b5ce7f72f30fc6923a0d15c888ee8226a9dc0e816b0958d5eb9736a14e9903/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/be73e6ee9cfbc523a16da6762a595cf146e53175108e94a7a95682a7a60f839a/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/73f7b786842097c557b0c6ca2ec51653f0bc5b4ee639bf0707d91ca2ecd0ab1e/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/2cab9a76247a30304fbca69975389f4acf035d4d14b9b481293192773154c0c0/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/7aefd3d9096ba85a1258d119c3e16784bbba6caa3e79c4e1a6075f4a0c9c6eaf/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/fe62d390e5f241cd02933d7fe4ce91ae213609f6d9572b2c00af7582bd5ce485/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/faa209385b2295c7a599f97b8e11114fa1a9e5ce45cc1e468c2a8775b6112e9a/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/83c5348a9cfb57a8b04c66c8ce9ecf0668f3d0278b5ef6742ab00ff8975617a9/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/24e4851ab9a07602f1e7e20481e9e99a13156a91b55826b43e19d57e839a69b2/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/5a89322b0503e6728b0a074f43fbbc62dd4418bcbf9ca2b40252e24d30dd06eb/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/249e2f1515d2f97a93612a3345c37eee9388ca63eae04f352f1fdab329f4b158/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/59919a062ccef95fa14a727c17a6e0fababd82264b16c78b3f27c4cadaabaec5/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/413d6a31814ffdb38ab5d7038303e8c177175b1bb5a9053134ae74a95b6b61a6/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/425b4f8e2d38fe19d9fcd8ea82a29a5076c64cdaf7ac82f21ddd21a7c3af6799/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/c406d2f527bf399f62f0232a8013d0e0c3577d988763f2ba580be88c32cd938e/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/e779b708b03f9794848bc14a1fabfb9c29526a374b92a773d81fe7e36e9fe5f6/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/7e50b9d162fd91a5ec2f41f17c2cff71364f5fc03430bbdd33146350aba2883f/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-449:{mountpoint:/var/lib/containers/storage/overlay/2eff5d27c2616824ba601df19fc19e2993460f949c9870361ff9a5c05dad2319/merged major:0 minor:449 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/412b4451216a2b71d1ef808d2e5d64f5607b2fee5c22c9c1c086e1b1d7ff4472/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/d307fcc9d2145006d57490fd3812348d22d2513dd622f948300a56b0a67ef907/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/cb19cc5560730babafbbb051a2338354ee169e9ec8ef55a12f42fdb786de7254/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-464:{mountpoint:/var/lib/containers/storage/overlay/e7548057a4ed404866a34c6799faabd7605ed28db6ea37140d5d249f4598dc0a/merged major:0 minor:464 fsType:overlay blockSize:0} overlay_0-469:{mountpoint:/var/lib/containers/storage/overlay/b0442326a47457b46ff8b63a3b8b7b4229806ec0a9ba53439607671a77fda28d/merged major:0 minor:469 fsType:overlay blockSize:0} overlay_0-475:{mountpoint:/var/lib/containers/storage/overlay/3c0945c0a50e8b2cf77d38f58d9c85498f67ff9e1c29aedfbd9cda18c4350e72/merged major:0 minor:475 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/057e7f59be5e59230036fcb290ab245703ee2c402ed1d8674a381caa57f42e13/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-485:{mountpoint:/var/lib/containers/storage/overlay/ca569e647c54cf1e66169721bac2be29ebd9092568af5bf67dedcc19e0358ec9/merged major:0 minor:485 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/864948eef615a1d5eafbf2560d048479cca01a3a7a4f4c19ba5f9bc6feb58885/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/eb61aec417cb98eb96b147bc748fe42a5a9697099fd9d7c31f6e3a6446877fae/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-507:{mountpoint:/var/lib/containers/storage/overlay/b9b55541a371516842e0b58510fae4d846659a00e4825caebba9104fd72b5d49/merged major:0 minor:507 fsType:overlay blockSize:0} overlay_0-509:{mountpoint:/var/lib/containers/storage/overlay/8c2dbb65c78537e173a956d749e10286e6641f6c18a498925e9bc19dc2e6cb03/merged major:0 minor:509 fsType:overlay blockSize:0} overlay_0-511:{mountpoint:/var/lib/containers/storage/overlay/2f258167dff0f601ca06f16ec652e4d8364844d845c674aa7de35aba9b8798a5/merged major:0 minor:511 fsType:overlay blockSize:0} overlay_0-513:{mountpoint:/var/lib/containers/storage/overlay/7beeb6e18a09747b90dd691209dad36a4d4f6ca5bc1bb0e289fcb321039d5ba0/merged major:0 minor:513 fsType:overlay blockSize:0} overlay_0-515:{mountpoint:/var/lib/containers/storage/overlay/dda7dcd280ae249e6eef8e230f6ec2376c5043079e4adf239632935014269b38/merged major:0 minor:515 fsType:overlay blockSize:0} overlay_0-521:{mountpoint:/var/lib/containers/storage/overlay/7845f8e64b5ccf8778e5344191bf1f68e77154931f880276fed6f4cf5d9e7b4c/merged major:0 minor:521 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/19b196a72ca879f63798c4c07568456eb633e15dd0c9bbaba5933b4731786801/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/77060ef3c1fb4ed07d877fce8670d9c22bfa289ae561e27474c43ba6c06b4c83/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ef49f2ee60c790ae7f567da62732bfa6fd36862e7d4d09b5fee0c3910bd41590/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/889d7bccc2e66764961cd07c95b2c382619f97ac624cec195aaebe04e6d4b404/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/a4525ab04a3a5c8af107ad893613ab30854fc325b7e5cd3dfb86b7a46d2b6c95/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/956997cb60a220081b7e60b5a3a24c2c39daea6fcfb80730cb3bcae1322c59d1/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/fabcff3039e54f817c01f3c68f569b42ee5f97ff355eee7879a68b2a0535079b/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/274e7277a658339c13e41cb129c76d817d74d9252c8459216e72d480cc99d905/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/f85e41a39e5b3869293290e1fb4a8b43365c73f8bdc596b92f42b433574af59b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/230cac340b73ad91a4fdc3deafa1022d2d2a89f26f80d78ad278bfcb28843d2f/merged major:0 minor:97 fsType:overlay blockSize:0}]
Nov 02 12:27:33.959166 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.958669    5879 manager.go:217] Machine: {Timestamp:2026-11-02 12:27:33.958170603 +0000 UTC m=+0.064873445 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100197 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25f5866a5f816750b37ff752a560c7 SystemUUID:ec25f586-6a5f-8167-50b3-7ff752a560c7 BootID:eebfa388-deb3-4929-9203-04310d8b1020 Filesystems:[{Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de9c9447ce3611f1658dd55296dfade31452a41ea352a1d821aafda26255bb78/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~projected/kube-api-access-lw7dg DeviceMajor:0 DeviceMinor:269 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/14492824-39e2-4e98-a91d-ac81382ad70f/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/718bc5cd-de70-4920-9e8f-5bc14fe1f45c/volumes/kubernetes.io~projected/kube-api-access-p8bmf DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b35a43d4116b66143b827d78db7d612bd73cfd3adfa90f575dab329f5fb440f6/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:254 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-509 DeviceMajor:0 DeviceMinor:509 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-511 DeviceMajor:0 DeviceMinor:511 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1827f1a3-a7e6-4d57-907a-763220ce3e0e/volumes/kubernetes.io~projected/kube-api-access-gxbqz DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:296 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:438 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:442 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/78221f6f-2223-4ac2-92da-b763ff2a6b39/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:495 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-515 DeviceMajor:0 DeviceMinor:515 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:267 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55b8e7e1aedc98e749e1eaba81690c1728619590490d37ce5ea814f260932c09/userdata/shm DeviceMajor:0 DeviceMinor:308 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:431 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff9623fb-ce84-42fc-aa52-fc263d66ec90/volumes/kubernetes.io~projected/kube-api-access-jmz9r DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:297 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c0067039-d47d-4ee4-8e02-1e24cdc26732/volumes/kubernetes.io~projected/kube-api-access-b6tkl DeviceMajor:0 DeviceMinor:348 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0162c6de-30e1-4e0b-949e-43b813b1e953/volumes/kubernetes.io~projected/kube-api-access-pv5cq DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:265 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-513 DeviceMajor:0 DeviceMinor:513 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69c6d5dffcb708513d462969fcde74577a883a833d0af76bbadf3a54581c7d81/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff9623fb-ce84-42fc-aa52-fc263d66ec90/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c/volumes/kubernetes.io~projected/kube-api-access-jjfhm DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:436 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:437 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-464 DeviceMajor:0 DeviceMinor:464 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/259d31ef-0ef7-4655-9121-88fd07468c38/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:499 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:266 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~projected/kube-api-access-g4gb2 DeviceMajor:0 DeviceMinor:372 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d7ceb6da9aa7950150360aef73ffba03976248108c7724d3f6320157a3b2a531/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/30c18bb65aa5a2520d2f13157f87e6701b10f0ca336dff3e800114f451bf5fcf/userdata/shm DeviceMajor:0 DeviceMinor:361 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:439 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5dc2fd5be2e223841f09bb68b1aed9a57c34a8bbafeda69531422cb6a987424/userdata/shm DeviceMajor:0 DeviceMinor:500 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/718bc5cd-de70-4920-9e8f-5bc14fe1f45c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d59c905e-0042-4777-a2fe-d5d5ae04e47d/volumes/kubernetes.io~projected/kube-api-access-vgpqc DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471/volumes/kubernetes.io~projected/kube-api-access-97wkz DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:293 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/14492824-39e2-4e98-a91d-ac81382ad70f/volumes/kubernetes.io~projected/kube-api-access-ts8bt DeviceMajor:0 DeviceMinor:341 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:370 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:369 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:371 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e27ad6ea38d49de9df8fd5a312d081f44a213d984789d22dbc95b2788a87fb46/userdata/shm DeviceMajor:0 DeviceMinor:447 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:444 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/157a29163cc443f325ee6ca2d4bb24477fe7d5c24fad4cb0a6bdbaefb07c52f7/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:264 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:441 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:441 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b72267e-e7f7-4115-af3e-14a382e659af/volumes/kubernetes.io~projected/kube-api-access-csd2s DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5927fee2edddf17965fe3e93cea6caa0de3abc9a67344b2da5770977d8c334d7/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d064abf07fa1a44346d0f42e8579fc1e71d70a91f06f29e04286221680a236d3/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f83efbb007fca102cf724127bf3e9d727ccf747a86eddefcd6ff5e20c45adaad/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7384673b07504ae5d5827772219deea4aa974c328d8347c76298f5bb4884961d/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:292 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:441 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-507 DeviceMajor:0 DeviceMinor:507 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c8202fc-395e-4188-af53-19a2d40a55fe/volumes/kubernetes.io~projected/kube-api-access-5p26c DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~projected/kube-api-access-76ddl DeviceMajor:0 DeviceMinor:298 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:266 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/14492824-39e2-4e98-a91d-ac81382ad70f/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:334 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:266 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca3d73812b549dbba9eec19c804667161aa5f128dfb36b0096e84230fe72d876/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-449 DeviceMajor:0 DeviceMinor:449 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-475 DeviceMajor:0 DeviceMinor:475 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ec459b10e653210fec5cc58b0d25625802d678b9f00870815f6600ee2e24a03/userdata/shm DeviceMajor:0 DeviceMinor:505 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a41b8e21-8e18-4e58-85d9-c24ba888b2f6/volumes/kubernetes.io~projected/kube-api-access-c5x28 DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8c1a1eb2cfe560549d2d35c957c0c252df53c896f45ae08a1e00315e363a389/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9215ec4-711e-400c-852c-aaaebbd7a7ec/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/718bc5cd-de70-4920-9e8f-5bc14fe1f45c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0cf6192d-d2e9-498b-b582-32305e1572d7/volumes/kubernetes.io~projected/kube-api-access-qh9vf DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-485 DeviceMajor:0 DeviceMinor:485 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba748218a38311cf4bf3a380834741cab72cab6a0a9c885f0a9a23cc6241f87c/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:255 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5dda63d8ced7535d751f9351f0ff65191cf54fa01ccad87dc71ecc3c4441493a/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/025ece4b-9534-475f-b96d-0cb3edfe3ac3/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:368 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:266 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:268 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:445 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-469 DeviceMajor:0 DeviceMinor:469 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b89c2be7-3500-4d9a-ac82-4ef367b83e3e/volumes/kubernetes.io~projected/kube-api-access-wvmb9 DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60/volumes/kubernetes.io~projected/kube-api-access-fwdjq DeviceMajor:0 DeviceMinor:502 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:254 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8971abcc6165396c2ba45e7697f7e03d011c252b25c8d99ca87d518dce60cf33/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~projected/kube-api-access-j4jzs DeviceMajor:0 DeviceMinor:446 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:441 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/775f38a882bf57ebbe1940016f33cd09377cf171a3cd556ea991ccad62807b4f/userdata/shm DeviceMajor:0 DeviceMinor:503 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7b8bfaaab6e0d96223a2d994677ddcd08d7d624ffa75453543f3dc158adaf4e/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf82277f49d8e3ca224e9eb33c605be11e0445d211276acc34b917b79b268979/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d9215ec4-711e-400c-852c-aaaebbd7a7ec/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:228 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-521 DeviceMajor:0 DeviceMinor:521 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db036f5ef9413c246a4091c304bee6732a34ae5a1d4a6c4352dfb0c18681e0a5/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14492824-39e2-4e98-a91d-ac81382ad70f/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c0067039-d47d-4ee4-8e02-1e24cdc26732/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:346 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/78221f6f-2223-4ac2-92da-b763ff2a6b39/volumes/kubernetes.io~projected/kube-api-access-p8dqb DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9b72267e-e7f7-4115-af3e-14a382e659af/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ff9623fb-ce84-42fc-aa52-fc263d66ec90/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:263 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c8202fc-395e-4188-af53-19a2d40a55fe/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d91a6143861ec9a16bae11c0ae9b49f872c5a9a0be4dbe6642c17e0123ec6ff/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:262 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:440 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:443 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a/volumes/kubernetes.io~projected/kube-api-access-xnwn9 DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d9215ec4-711e-400c-852c-aaaebbd7a7ec/volumes/kubernetes.io~projected/kube-api-access-hx6p7 DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/851cf5aa6fbc6747bee11fcc8c028d936ed2f893311ae001980a6a22cf92103b/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:30c18bb65aa5a25 MacAddress:e2:7b:5f:07:60:a8 Speed:10000 Mtu:8901} {Name:4ec459b10e65321 MacAddress:92:47:6b:cd:de:3a Speed:10000 Mtu:8901} {Name:55b8e7e1aedc98e MacAddress:26:2c:dd:4a:32:34 Speed:10000 Mtu:8901} {Name:5927fee2edddf17 MacAddress:02:ce:94:49:9a:70 Speed:10000 Mtu:8901} {Name:5dda63d8ced7535 MacAddress:86:a8:29:84:14:c9 Speed:10000 Mtu:8901} {Name:775f38a882bf57e MacAddress:52:94:40:66:7d:be Speed:10000 Mtu:8901} {Name:851cf5aa6fbc674 MacAddress:8e:76:2a:a8:fe:5c Speed:10000 Mtu:8901} {Name:8971abcc6165396 MacAddress:a2:fd:97:5a:92:02 Speed:10000 Mtu:8901} {Name:b35a43d4116b661 MacAddress:1e:18:b0:7c:37:29 Speed:10000 Mtu:8901} {Name:b5dc2fd5be2e223 MacAddress:72:43:35:00:a0:f0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fa:81:df:3d Speed:0 Mtu:9001} {Name:br-int MacAddress:62:fd:83:f3:d5:95 Speed:0 Mtu:8901} {Name:d7ceb6da9aa7950 MacAddress:0e:10:08:ab:2d:48 Speed:10000 Mtu:8901} {Name:db036f5ef9413c2 MacAddress:3a:e9:f9:a5:97:cb Speed:10000 Mtu:8901} {Name:de9c9447ce3611f MacAddress:5a:88:cd:ac:cd:2a Speed:10000 Mtu:8901} {Name:e27ad6ea38d49de MacAddress:76:37:3b:2d:b8:67 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fa:81:df:3d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b6:a9:b6:d7:9e:99 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1a:d7:cf:64:08:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 12:27:33.959478 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.959170    5879 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 12:27:33.959478 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.959221    5879 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 12:27:33.959642 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.959608    5879 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 12:27:33.959788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.959643    5879 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-229.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 12:27:33.959829 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.959791    5879 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 12:27:33.959829 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.959799    5879 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 12:27:33.959829 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.959810    5879 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 12:27:33.959913 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.959830    5879 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 12:27:33.959942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.959925    5879 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 12:27:33.960157 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.960147    5879 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 12:27:33.960277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.960269    5879 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 12:27:33.960317 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.960308    5879 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 12:27:33.960354 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.960327    5879 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 12:27:33.960354 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.960348    5879 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 12:27:33.960417 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.960357    5879 kubelet.go:405] "Adding apiserver pod source"
Nov 02 12:27:33.960417 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.960369    5879 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 12:27:33.961493 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.961476    5879 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 12:27:33.961493 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.961489    5879 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 12:27:33.962180 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.962104    5879 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 12:27:33.965093 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965077    5879 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 12:27:33.965093 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965090    5879 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 12:27:33.965196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965191    5879 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 12:27:33.965403 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965391    5879 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 12:27:33.965434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965421    5879 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 12:27:33.965521 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965512    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 12:27:33.965558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965526    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 12:27:33.965558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965532    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 12:27:33.965558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965538    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 12:27:33.965558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965543    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 12:27:33.965558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965548    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 12:27:33.965558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965554    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 12:27:33.965558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965559    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 12:27:33.965796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965567    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 12:27:33.965796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965573    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 12:27:33.965796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965588    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 12:27:33.965796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965596    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 12:27:33.965796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965619    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 12:27:33.965796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.965624    5879 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 12:27:33.969053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.969040    5879 server.go:1263] "Started kubelet"
Nov 02 12:27:33.969125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.969101    5879 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 12:27:33.969703 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.969167    5879 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 12:27:33.969703 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.969248    5879 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 12:27:33.969621 ip-10-0-134-229 systemd[1]: Started Kubernetes Kubelet.
Nov 02 12:27:33.971335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.971286    5879 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 12:27:33.972553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.972537    5879 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 12:27:33.976303 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.976285    5879 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 12:27:33.976476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.976452    5879 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:33.976649    5879 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-229.ec2.internal\" not found"
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.976703    5879 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.976710    5879 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.976910    5879 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.977416    5879 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 12:21:24 +0000 UTC" deadline="2028-05-05 03:43:04.29645571 +0000 UTC"
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.977442    5879 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13191h15m30.319017321s"
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.977868    5879 factory.go:55] Registering systemd factory
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.977890    5879 factory.go:223] Registration of the systemd container factory successfully
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.978226    5879 factory.go:153] Registering CRI-O factory
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.978237    5879 factory.go:223] Registration of the crio container factory successfully
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.978283    5879 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.978298    5879 factory.go:103] Registering Raw factory
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.978310    5879 manager.go:1199] Started watching for new ooms in manager
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.978667    5879 manager.go:319] Starting recovery of all containers
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980081    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14492824-39e2-4e98-a91d-ac81382ad70f" volumeName="kubernetes.io/empty-dir/14492824-39e2-4e98-a91d-ac81382ad70f-audit-log" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980136    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-cluster-tls-config" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980160    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a41b8e21-8e18-4e58-85d9-c24ba888b2f6" volumeName="kubernetes.io/projected/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-kube-api-access-c5x28" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980175    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9215ec4-711e-400c-852c-aaaebbd7a7ec" volumeName="kubernetes.io/configmap/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980192    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980209    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0162c6de-30e1-4e0b-949e-43b813b1e953" volumeName="kubernetes.io/configmap/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-daemon-config" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980226    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0067039-d47d-4ee4-8e02-1e24cdc26732" volumeName="kubernetes.io/projected/c0067039-d47d-4ee4-8e02-1e24cdc26732-kube-api-access-b6tkl" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980243    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d59c905e-0042-4777-a2fe-d5d5ae04e47d" volumeName="kubernetes.io/projected/d59c905e-0042-4777-a2fe-d5d5ae04e47d-kube-api-access-vgpqc" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980261    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980276    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980297    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-config-out" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980314    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bb243e5-8e99-4391-9a55-6e3a8f82ae0c" volumeName="kubernetes.io/secret/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980329    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a" volumeName="kubernetes.io/configmap/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980367    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff9623fb-ce84-42fc-aa52-fc263d66ec90" volumeName="kubernetes.io/projected/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-api-access-jmz9r" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980388    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980402    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471" volumeName="kubernetes.io/secret/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980421    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-volume" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980435    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0162c6de-30e1-4e0b-949e-43b813b1e953" volumeName="kubernetes.io/projected/0162c6de-30e1-4e0b-949e-43b813b1e953-kube-api-access-pv5cq" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980452    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="025ece4b-9534-475f-b96d-0cb3edfe3ac3" volumeName="kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980469    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14492824-39e2-4e98-a91d-ac81382ad70f" volumeName="kubernetes.io/projected/14492824-39e2-4e98-a91d-ac81382ad70f-kube-api-access-ts8bt" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980485    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14492824-39e2-4e98-a91d-ac81382ad70f" volumeName="kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980502    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471" volumeName="kubernetes.io/projected/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-kube-api-access-97wkz" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980521    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c8202fc-395e-4188-af53-19a2d40a55fe" volumeName="kubernetes.io/secret/4c8202fc-395e-4188-af53-19a2d40a55fe-cert" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980540    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14492824-39e2-4e98-a91d-ac81382ad70f" volumeName="kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-secret-metrics-server-tls" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980557    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72" volumeName="kubernetes.io/empty-dir/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-tmp" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980570    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471" volumeName="kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovnkube-config" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980587    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a" volumeName="kubernetes.io/projected/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-kube-api-access-xnwn9" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980618    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff9623fb-ce84-42fc-aa52-fc263d66ec90" volumeName="kubernetes.io/configmap/ff9623fb-ce84-42fc-aa52-fc263d66ec90-metrics-client-ca" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980643    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-tls-assets" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980658    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-grpc-tls" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980671    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="025ece4b-9534-475f-b96d-0cb3edfe3ac3" volumeName="kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-federate-client-tls" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980683    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cf6192d-d2e9-498b-b582-32305e1572d7" volumeName="kubernetes.io/projected/0cf6192d-d2e9-498b-b582-32305e1572d7-kube-api-access-qh9vf" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980696    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bb243e5-8e99-4391-9a55-6e3a8f82ae0c" volumeName="kubernetes.io/secret/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980710    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980725    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980738    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="259d31ef-0ef7-4655-9121-88fd07468c38" volumeName="kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980751    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-metrics-client-ca" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980764    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9215ec4-711e-400c-852c-aaaebbd7a7ec" volumeName="kubernetes.io/secret/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980776    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980788    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="18c61308-36a9-4eb4-9cc9-ac85bf1bdd60" volumeName="kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980801    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a" volumeName="kubernetes.io/empty-dir/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-data-volume" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980814    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83dcdc7a-f1ba-4570-bed5-a40b0d3b213d" volumeName="kubernetes.io/configmap/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d-konnectivity-ca" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980826    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff9623fb-ce84-42fc-aa52-fc263d66ec90" volumeName="kubernetes.io/secret/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980839    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14492824-39e2-4e98-a91d-ac81382ad70f" volumeName="kubernetes.io/configmap/14492824-39e2-4e98-a91d-ac81382ad70f-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980852    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14492824-39e2-4e98-a91d-ac81382ad70f" volumeName="kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-client-ca-bundle" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980866    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980880    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980893    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="025ece4b-9534-475f-b96d-0cb3edfe3ac3" volumeName="kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-metrics-client-ca" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980908    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1827f1a3-a7e6-4d57-907a-763220ce3e0e" volumeName="kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980924    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72" volumeName="kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980937    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b89c2be7-3500-4d9a-ac82-4ef367b83e3e" volumeName="kubernetes.io/projected/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-kube-api-access-wvmb9" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980950    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72" volumeName="kubernetes.io/configmap/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-metrics-client-ca" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980964    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9215ec4-711e-400c-852c-aaaebbd7a7ec" volumeName="kubernetes.io/projected/d9215ec4-711e-400c-852c-aaaebbd7a7ec-kube-api-access-hx6p7" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.980977    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b72267e-e7f7-4115-af3e-14a382e659af" volumeName="kubernetes.io/configmap/9b72267e-e7f7-4115-af3e-14a382e659af-config-volume" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981008    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0162c6de-30e1-4e0b-949e-43b813b1e953" volumeName="kubernetes.io/configmap/0162c6de-30e1-4e0b-949e-43b813b1e953-cni-binary-copy" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981022    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="025ece4b-9534-475f-b96d-0cb3edfe3ac3" volumeName="kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-secret-telemeter-client" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981044    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14492824-39e2-4e98-a91d-ac81382ad70f" volumeName="kubernetes.io/configmap/14492824-39e2-4e98-a91d-ac81382ad70f-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981056    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72" volumeName="kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 12:27:33.981018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981071    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-out" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981084    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1827f1a3-a7e6-4d57-907a-763220ce3e0e" volumeName="kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cni-binary-copy" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981096    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72" volumeName="kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981110    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="025ece4b-9534-475f-b96d-0cb3edfe3ac3" volumeName="kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-telemeter-client-tls" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981121    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72" volumeName="kubernetes.io/projected/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-kube-api-access-76ddl" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981136    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78221f6f-2223-4ac2-92da-b763ff2a6b39" volumeName="kubernetes.io/projected/78221f6f-2223-4ac2-92da-b763ff2a6b39-kube-api-access-p8dqb" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981153    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="025ece4b-9534-475f-b96d-0cb3edfe3ac3" volumeName="kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981165    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72" volumeName="kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981177    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bb243e5-8e99-4391-9a55-6e3a8f82ae0c" volumeName="kubernetes.io/configmap/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-metrics-client-ca" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981189    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981202    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b72267e-e7f7-4115-af3e-14a382e659af" volumeName="kubernetes.io/empty-dir/9b72267e-e7f7-4115-af3e-14a382e659af-tmp-dir" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981214    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-web-config" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981228    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-db" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981240    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-main-db" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981252    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-web-config" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981263    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="025ece4b-9534-475f-b96d-0cb3edfe3ac3" volumeName="kubernetes.io/projected/025ece4b-9534-475f-b96d-0cb3edfe3ac3-kube-api-access-g4gb2" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981283    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981311    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="718bc5cd-de70-4920-9e8f-5bc14fe1f45c" volumeName="kubernetes.io/empty-dir/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-tmp" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981324    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1827f1a3-a7e6-4d57-907a-763220ce3e0e" volumeName="kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981340    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff9623fb-ce84-42fc-aa52-fc263d66ec90" volumeName="kubernetes.io/empty-dir/ff9623fb-ce84-42fc-aa52-fc263d66ec90-volume-directive-shadow" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981353    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981371    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78221f6f-2223-4ac2-92da-b763ff2a6b39" volumeName="kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981384    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-kube-api-access-j4jzs" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981397    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-config" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981410    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bb243e5-8e99-4391-9a55-6e3a8f82ae0c" volumeName="kubernetes.io/projected/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-kube-api-access-jjfhm" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981423    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-kube-api-access-lw7dg" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981435    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="718bc5cd-de70-4920-9e8f-5bc14fe1f45c" volumeName="kubernetes.io/projected/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-kube-api-access-p8bmf" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981448    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9215ec4-711e-400c-852c-aaaebbd7a7ec" volumeName="kubernetes.io/empty-dir/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-textfile" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981461    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981474    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b89c2be7-3500-4d9a-ac82-4ef367b83e3e" volumeName="kubernetes.io/empty-dir/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-tmp-dir" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981488    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="025ece4b-9534-475f-b96d-0cb3edfe3ac3" volumeName="kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981500    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a" volumeName="kubernetes.io/secret/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981512    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b72267e-e7f7-4115-af3e-14a382e659af" volumeName="kubernetes.io/projected/9b72267e-e7f7-4115-af3e-14a382e659af-kube-api-access-csd2s" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981526    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a41b8e21-8e18-4e58-85d9-c24ba888b2f6" volumeName="kubernetes.io/configmap/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-iptables-alerter-script" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981540    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9215ec4-711e-400c-852c-aaaebbd7a7ec" volumeName="kubernetes.io/configmap/d9215ec4-711e-400c-852c-aaaebbd7a7ec-metrics-client-ca" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981554    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981567    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981582    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c8202fc-395e-4188-af53-19a2d40a55fe" volumeName="kubernetes.io/projected/4c8202fc-395e-4188-af53-19a2d40a55fe-kube-api-access-5p26c" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981594    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="718bc5cd-de70-4920-9e8f-5bc14fe1f45c" volumeName="kubernetes.io/empty-dir/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-tuned" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981607    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b72267e-e7f7-4115-af3e-14a382e659af" volumeName="kubernetes.io/secret/9b72267e-e7f7-4115-af3e-14a382e659af-metrics-tls" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981620    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d59c905e-0042-4777-a2fe-d5d5ae04e47d" volumeName="kubernetes.io/configmap/d59c905e-0042-4777-a2fe-d5d5ae04e47d-serviceca" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981632    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981645    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471" volumeName="kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981658    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e92edc-76ab-4aac-9928-4fd21b20b7e8" volumeName="kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-tls-assets" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981671    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83dcdc7a-f1ba-4570-bed5-a40b0d3b213d" volumeName="kubernetes.io/secret/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d-agent-certs" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981684    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72" volumeName="kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-grpc-tls" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981695    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff9623fb-ce84-42fc-aa52-fc263d66ec90" volumeName="kubernetes.io/secret/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-tls" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981709    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1827f1a3-a7e6-4d57-907a-763220ce3e0e" volumeName="kubernetes.io/projected/1827f1a3-a7e6-4d57-907a-763220ce3e0e-kube-api-access-gxbqz" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981723    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72" volumeName="kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981736    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471" volumeName="kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-env-overrides" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981748    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0067039-d47d-4ee4-8e02-1e24cdc26732" volumeName="kubernetes.io/secret/c0067039-d47d-4ee4-8e02-1e24cdc26732-monitoring-plugin-cert" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981762    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d9215ec4-711e-400c-852c-aaaebbd7a7ec" volumeName="kubernetes.io/secret/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-tls" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981776    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff9623fb-ce84-42fc-aa52-fc263d66ec90" volumeName="kubernetes.io/configmap/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981789    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981820    5879 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffb2d74e-5ea9-48e3-98b8-53963e608786" volumeName="kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-metrics-client-certs" seLinuxMountContext=""
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981833    5879 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 12:27:33.984480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:33.981841    5879 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 12:27:34.007079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.007033    5879 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 12:27:34.008589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.008564    5879 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 12:27:34.008589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.008590    5879 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 12:27:34.008758 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.008612    5879 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 12:27:34.008758 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:34.008653    5879 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 12:27:34.021696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.021647    5879 generic.go:356] "Generic (PLEG): container finished" podID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerID="8cc3c8ee3ebb3173b9b413353bd7104ed4d37655ff2e43588b51ebc2df7b728c" exitCode=0
Nov 02 12:27:34.034256 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.034234    5879 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 12:27:34.035583 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.035557    5879 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:27:34.040810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.040787    5879 generic.go:356] "Generic (PLEG): container finished" podID="1827f1a3-a7e6-4d57-907a-763220ce3e0e" containerID="6a71f99776bf44387c17651af19a6d32c358d439ffc5b12d53e97d9f291079b5" exitCode=0
Nov 02 12:27:34.040810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.040807    5879 generic.go:356] "Generic (PLEG): container finished" podID="1827f1a3-a7e6-4d57-907a-763220ce3e0e" containerID="b9e91cc4b5d14290085fc725e1e5226eac7e09114688b928e8908e8e651ba1a2" exitCode=0
Nov 02 12:27:34.040810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.040813    5879 generic.go:356] "Generic (PLEG): container finished" podID="1827f1a3-a7e6-4d57-907a-763220ce3e0e" containerID="2504f59d39bceafc59e33532879d30dfc710104740c4425e1b4e66b50e522510" exitCode=0
Nov 02 12:27:34.041073 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.040819    5879 generic.go:356] "Generic (PLEG): container finished" podID="1827f1a3-a7e6-4d57-907a-763220ce3e0e" containerID="039d443308fceef2ae8f242348a0e91a4fbb695a384a5ad2142d30f524d8e5d5" exitCode=0
Nov 02 12:27:34.041073 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.040824    5879 generic.go:356] "Generic (PLEG): container finished" podID="1827f1a3-a7e6-4d57-907a-763220ce3e0e" containerID="9b0a6f1d53dfd70d04bd5b3a4611244d916ca297d6941b23805c2f447f62234d" exitCode=0
Nov 02 12:27:34.041073 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.040829    5879 generic.go:356] "Generic (PLEG): container finished" podID="1827f1a3-a7e6-4d57-907a-763220ce3e0e" containerID="e46ac257fe9cfc77cb6df5163ad486e4b9e81f17356d70104250826bfa67bafb" exitCode=0
Nov 02 12:27:34.047828 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.047804    5879 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:27:34.048204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.048179    5879 generic.go:356] "Generic (PLEG): container finished" podID="55b1ac54009e3b5ad6df9f2e917c2d8b" containerID="0cab7c354846ab96475eea5db8af5c1fbb57fa56d6e6da995b0abe47ec578798" exitCode=0
Nov 02 12:27:34.052073 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.052048    5879 generic.go:356] "Generic (PLEG): container finished" podID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerID="e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64" exitCode=0
Nov 02 12:27:34.054326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.054305    5879 generic.go:356] "Generic (PLEG): container finished" podID="d9215ec4-711e-400c-852c-aaaebbd7a7ec" containerID="0cf71450c60d2a04679a5649c48299e93443c80652c7232a92dfab707d501c4c" exitCode=0
Nov 02 12:27:34.060674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.060649    5879 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:27:34.062203 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.062163    5879 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 12:27:34.108726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.108663    5879 manager.go:324] Recovery completed
Nov 02 12:27:34.108852 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:34.108759    5879 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 02 12:27:34.135007 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.134974    5879 cpu_manager.go:226] "Starting" policy="none"
Nov 02 12:27:34.135007 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.135006    5879 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 12:27:34.135233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.135027    5879 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 12:27:34.135233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.135178    5879 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 12:27:34.135233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.135191    5879 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 12:27:34.135233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.135216    5879 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 12:27:34.135233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.135224    5879 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 12:27:34.135233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.135231    5879 policy_none.go:50] "Start"
Nov 02 12:27:34.135513 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.135244    5879 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 12:27:34.135513 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.135257    5879 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 12:27:34.135513 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.135435    5879 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 12:27:34.135513 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.135445    5879 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 12:27:34.135513 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.135454    5879 policy_none.go:44] "Start"
Nov 02 12:27:34.140270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.140240    5879 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 12:27:34.142969 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.142954    5879 manager.go:341] "Starting Device Plugin manager"
Nov 02 12:27:34.143443 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:34.142980    5879 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 12:27:34.143512 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.143449    5879 server.go:87] "Starting device plugin registration server"
Nov 02 12:27:34.143708 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.143698    5879 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 12:27:34.143742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.143709    5879 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 12:27:34.143820 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.143802    5879 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 12:27:34.143925 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.143899    5879 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 12:27:34.144006 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.143938    5879 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 12:27:34.144006 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.143945    5879 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 12:27:34.144131 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.144036    5879 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T12:27:34.143923884Z","UUID":"fb459df0-1bca-48c8-af09-8a809a499300","Handler":null,"Name":"","Endpoint":""}
Nov 02 12:27:34.145337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.145320    5879 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 12:27:34.145459 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.145440    5879 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 12:27:34.145614 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:34.145562    5879 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 12:27:34.244234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.244202    5879 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 12:27:34.246248 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.246229    5879 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 12:27:34.246375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.246267    5879 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 12:27:34.246375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.246279    5879 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-229.ec2.internal" event="NodeHasSufficientPID"
Nov 02 12:27:34.246375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.246325    5879 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.286704 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.286671    5879 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.286871 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.286760    5879 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.309774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.309707    5879 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"]
Nov 02 12:27:34.310434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.310399    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal" event={"ID":"55b1ac54009e3b5ad6df9f2e917c2d8b","Type":"ContainerStarted","Data":"04875313b26af7d60c807628d6c07f10e95f95d802d12d26ee6f1094c3e01202"}
Nov 02 12:27:34.310526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.310439    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal" event={"ID":"55b1ac54009e3b5ad6df9f2e917c2d8b","Type":"ContainerDied","Data":"0cab7c354846ab96475eea5db8af5c1fbb57fa56d6e6da995b0abe47ec578798"}
Nov 02 12:27:34.310526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.310446    5879 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.310526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.310454    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal" event={"ID":"55b1ac54009e3b5ad6df9f2e917c2d8b","Type":"ContainerStarted","Data":"b7b8bfaaab6e0d96223a2d994677ddcd08d7d624ffa75453543f3dc158adaf4e"}
Nov 02 12:27:34.310526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.310502    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal" event={"ID":"f2aebf6f2caaa3c2095ff6e33664045c","Type":"ContainerStarted","Data":"682862ac54cd5f3423801db9cac745106ed992612a6e7e4dd13aba7fc8c3ef28"}
Nov 02 12:27:34.310526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.310515    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal" event={"ID":"f2aebf6f2caaa3c2095ff6e33664045c","Type":"ContainerStarted","Data":"7384673b07504ae5d5827772219deea4aa974c328d8347c76298f5bb4884961d"}
Nov 02 12:27:34.310775 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.310689    5879 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.339979 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.339950    5879 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:27:34.340137 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:34.340024    5879 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-229.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.340229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.340215    5879 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:27:34.340263 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:34.340252    5879 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.437066 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.437032    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2aebf6f2caaa3c2095ff6e33664045c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-229.ec2.internal\" (UID: \"f2aebf6f2caaa3c2095ff6e33664045c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.437066 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.437067    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55b1ac54009e3b5ad6df9f2e917c2d8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" (UID: \"55b1ac54009e3b5ad6df9f2e917c2d8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.437260 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.437087    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55b1ac54009e3b5ad6df9f2e917c2d8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" (UID: \"55b1ac54009e3b5ad6df9f2e917c2d8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.537660 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.537606    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2aebf6f2caaa3c2095ff6e33664045c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-229.ec2.internal\" (UID: \"f2aebf6f2caaa3c2095ff6e33664045c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.537660 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.537663    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55b1ac54009e3b5ad6df9f2e917c2d8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" (UID: \"55b1ac54009e3b5ad6df9f2e917c2d8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.537865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.537687    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55b1ac54009e3b5ad6df9f2e917c2d8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" (UID: \"55b1ac54009e3b5ad6df9f2e917c2d8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.537865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.537723    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f2aebf6f2caaa3c2095ff6e33664045c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-229.ec2.internal\" (UID: \"f2aebf6f2caaa3c2095ff6e33664045c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.537865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.537731    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55b1ac54009e3b5ad6df9f2e917c2d8b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" (UID: \"55b1ac54009e3b5ad6df9f2e917c2d8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.537865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.537731    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55b1ac54009e3b5ad6df9f2e917c2d8b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" (UID: \"55b1ac54009e3b5ad6df9f2e917c2d8b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:34.961237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:34.961204    5879 apiserver.go:52] "Watching apiserver"
Nov 02 12:27:35.020451 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.020413    5879 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 12:27:35.021191 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.021163    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lbm4n","openshift-dns/dns-default-snnjx","openshift-insights/insights-runtime-extractor-8mqtc","openshift-monitoring/thanos-querier-57f985446f-hkswt","openshift-image-registry/node-ca-mdzbn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal","openshift-monitoring/telemeter-client-cf7f64998-5wcsl","openshift-ovn-kubernetes/ovnkube-node-l9dwk","openshift-dns/node-resolver-k49jr","openshift-network-operator/iptables-alerter-c4mfn","kube-system/konnectivity-agent-26p8v","openshift-monitoring/alertmanager-main-0","openshift-monitoring/prometheus-k8s-0","kube-system/global-pull-secret-syncer-x7jpb","openshift-network-diagnostics/network-check-target-2hzq7","openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm","openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx","openshift-multus/network-metrics-daemon-2vkgz","kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal","openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr","openshift-ingress-canary/ingress-canary-nkq67","openshift-monitoring/metrics-server-f7db9d655-n8z5r","openshift-multus/multus-additional-cni-plugins-fspl8","openshift-cluster-node-tuning-operator/tuned-9w9rr","openshift-multus/multus-5qqx4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"]
Nov 02 12:27:35.039027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.038980    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 12:27:35.039027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039031    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 12:27:35.039296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039069    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 12:27:35.039296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039121    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:27:35.039296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039134    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 12:27:35.039296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039134    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 12:27:35.039296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039194    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 12:27:35.039480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039459    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fhvpr\""
Nov 02 12:27:35.039587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039505    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gr8bh\""
Nov 02 12:27:35.039724 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039704    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 12:27:35.039788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039705    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 12:27:35.039788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039743    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7btt2\""
Nov 02 12:27:35.039788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039750    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hrc62\""
Nov 02 12:27:35.039788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039772    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 12:27:35.039955 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039811    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 12:27:35.039955 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039833    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 12:27:35.039955 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039841    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 12:27:35.039955 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039875    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2lqbm\""
Nov 02 12:27:35.039955 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.039907    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 12:27:35.040163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.040058    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 12:27:35.040163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.040068    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vg52x\""
Nov 02 12:27:35.040163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.040068    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 12:27:35.040163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.040059    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 12:27:35.040163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.040141    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-scnn5\""
Nov 02 12:27:35.055852 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.055825    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-89s57\""
Nov 02 12:27:35.057589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.057572    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 12:27:35.057912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.057886    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 12:27:35.057912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.057895    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 12:27:35.058106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.057930    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 12:27:35.058106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058033    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 12:27:35.058106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058037    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 12:27:35.058264 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058247    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 12:27:35.058312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058280    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 12:27:35.058312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058292    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 12:27:35.058592 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058576    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 12:27:35.058734 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058721    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 12:27:35.058797 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058764    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 12:27:35.058883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058868    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 12:27:35.058933 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058922    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 12:27:35.058933 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058927    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 12:27:35.059041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058974    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 12:27:35.059041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058984    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 12:27:35.059041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.058974    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 12:27:35.059041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.059009    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 12:27:35.059213 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.059048    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 12:27:35.059260 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.059237    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 12:27:35.059260 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.059256    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 12:27:35.059340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.059277    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 12:27:35.059340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.059281    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 02 12:27:35.059434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.059355    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 12:27:35.059434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.059372    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 12:27:35.059434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.059389    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 12:27:35.059434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.059416    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:27:35.059434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.059423    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 12:27:35.073164 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.073096    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 12:27:35.073433 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.073413    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 12:27:35.073529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.073451    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 12:27:35.073529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.073456    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 12:27:35.073529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.073457    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 12:27:35.073794 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.073776    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 12:27:35.073892 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.073816    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 12:27:35.073892 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.073880    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 12:27:35.074029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074016    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 12:27:35.074110 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074075    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 12:27:35.074190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074167    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 12:27:35.074258 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074208    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 12:27:35.074258 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074232    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-g8ktw\""
Nov 02 12:27:35.074364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074295    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 02 12:27:35.074364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074315    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 12:27:35.074364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074358    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 12:27:35.074500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074351    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 12:27:35.074500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074426    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 12:27:35.074500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074391    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 12:27:35.074500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074421    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 12:27:35.074652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074511    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 12:27:35.074652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074529    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bbkt9\""
Nov 02 12:27:35.074652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074552    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 12:27:35.074652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074568    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 12:27:35.074652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074591    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 12:27:35.074652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074622    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 12:27:35.074869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074714    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 02 12:27:35.074869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074799    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l5rh2\""
Nov 02 12:27:35.074945 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.074876    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 12:27:35.075141 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.075103    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 12:27:35.075219 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.075157    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5bwmg\""
Nov 02 12:27:35.075219 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.075167    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l6cfl\""
Nov 02 12:27:35.075361 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.075321    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 12:27:35.075767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.075746    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-tlsnx\""
Nov 02 12:27:35.075986 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.075965    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 12:27:35.076156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076132    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-wm9fw\""
Nov 02 12:27:35.076273 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076254    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mbf6h\""
Nov 02 12:27:35.076384 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076363    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 12:27:35.076456 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076432    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2bc7v\""
Nov 02 12:27:35.076456 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076435    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4xfxz\""
Nov 02 12:27:35.076549 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076499    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 12:27:35.076549 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076521    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 12:27:35.076658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076579    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-l6knz\""
Nov 02 12:27:35.076658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076606    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 12:27:35.076658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076639    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-67u7r1mgu2vji\""
Nov 02 12:27:35.076658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076651    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 12:27:35.076831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076661    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rbwvz\""
Nov 02 12:27:35.076831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076692    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c8tgcn5qluvqu\""
Nov 02 12:27:35.076831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076709    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 02 12:27:35.076831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076749    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 12:27:35.076831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076682    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pd4kn\""
Nov 02 12:27:35.076831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076764    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 12:27:35.076831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076770    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-v8788\""
Nov 02 12:27:35.076831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076751    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 12:27:35.076831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076814    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 12:27:35.077280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.076885    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 12:27:35.077280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.077014    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7sfkq\""
Nov 02 12:27:35.077280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.077166    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 12:27:35.077280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.077208    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-g5vwz\""
Nov 02 12:27:35.077481 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.077312    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 12:27:35.077585 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.077570    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-gnmlss15lorj\""
Nov 02 12:27:35.079414 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.079400    5879 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 12:27:35.085019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.084976    5879 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:35.085249 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.085227    5879 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:35.085544 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.085528    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 12:27:35.087356 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.087333    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 12:27:35.109466 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.109436    5879 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:27:35.109585 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:35.109499    5879 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-229.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:35.110068 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.110052    5879 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 12:27:35.110117 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:35.110104    5879 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-229.ec2.internal"
Nov 02 12:27:35.140081 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140049    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p26c\" (UniqueName: \"kubernetes.io/projected/4c8202fc-395e-4188-af53-19a2d40a55fe-kube-api-access-5p26c\") pod \"ingress-canary-nkq67\" (UID: \"4c8202fc-395e-4188-af53-19a2d40a55fe\") " pod="openshift-ingress-canary/ingress-canary-nkq67"
Nov 02 12:27:35.140246 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140093    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/259d31ef-0ef7-4655-9121-88fd07468c38-dbus\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:35.140246 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140121    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.140246 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140171    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.140246 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140227    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-etc-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.140461 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140324    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-tls\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.140461 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140377    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tkl\" (UniqueName: \"kubernetes.io/projected/c0067039-d47d-4ee4-8e02-1e24cdc26732-kube-api-access-b6tkl\") pod \"monitoring-plugin-77f4f76594-5d9mm\" (UID: \"c0067039-d47d-4ee4-8e02-1e24cdc26732\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:35.140461 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140414    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.140461 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140441    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-federate-client-tls\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.140461 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140454    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.140702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140466    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-env-overrides\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.140702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140460    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.140702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140493    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-tmp\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.140702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140670    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-tls\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.140702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140689    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-federate-client-tls\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.140930 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140705    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-env-overrides\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.140930 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140728    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-out\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.140930 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140747    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-tmp\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.140930 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140808    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-out\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.140930 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140823    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.140930 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140855    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.140930 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140881    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.140930 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140910    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-modprobe-d\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140928    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140939    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgpqc\" (UniqueName: \"kubernetes.io/projected/d59c905e-0042-4777-a2fe-d5d5ae04e47d-kube-api-access-vgpqc\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.140965    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwdjq\" (UniqueName: \"kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq\") pod \"network-check-target-2hzq7\" (UID: \"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60\") " pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141014    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-system-cni-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141041    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141083    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141088    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-client-ca-bundle\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141110    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141121    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141128    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-daemon-config\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141160    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-root\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141189    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141213    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-hosts-file\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141237    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141265    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-grpc-tls\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141292    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-daemon-config\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.141335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141299    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141344    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-client-ca-bundle\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141361    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-systemd-units\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141549    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141664    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-grpc-tls\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141664    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141720    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-systemd\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141760    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0162c6de-30e1-4e0b-949e-43b813b1e953-cni-binary-copy\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141784    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141786    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-registration-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141834    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw7dg\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-kube-api-access-lw7dg\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.141968    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0162c6de-30e1-4e0b-949e-43b813b1e953-cni-binary-copy\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142055    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-tmp\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142063    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cni-binary-copy\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142095    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.142163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142129    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-tmp\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142170    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9215ec4-711e-400c-852c-aaaebbd7a7ec-metrics-client-ca\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142227    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142280    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-sys-fs\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142340    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142379    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142412    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9215ec4-711e-400c-852c-aaaebbd7a7ec-metrics-client-ca\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142416    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142442    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142444    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142460    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97wkz\" (UniqueName: \"kubernetes.io/projected/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-kube-api-access-97wkz\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142477    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c8202fc-395e-4188-af53-19a2d40a55fe-cert\") pod \"ingress-canary-nkq67\" (UID: \"4c8202fc-395e-4188-af53-19a2d40a55fe\") " pod="openshift-ingress-canary/ingress-canary-nkq67"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142524    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnwn9\" (UniqueName: \"kubernetes.io/projected/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-kube-api-access-xnwn9\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142595    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142628    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142658    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142686    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4c8202fc-395e-4188-af53-19a2d40a55fe-cert\") pod \"ingress-canary-nkq67\" (UID: \"4c8202fc-395e-4188-af53-19a2d40a55fe\") " pod="openshift-ingress-canary/ingress-canary-nkq67"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142741    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-cni-multus\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.142864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142774    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-accelerators-collector-config\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142872    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.142921    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-web-config\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143029    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-accelerators-collector-config\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143054    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-log-socket\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143079    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143096    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-secret-metrics-server-tls\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143112    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts8bt\" (UniqueName: \"kubernetes.io/projected/14492824-39e2-4e98-a91d-ac81382ad70f-kube-api-access-ts8bt\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143115    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/259d31ef-0ef7-4655-9121-88fd07468c38-original-pull-secret\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143117    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143133    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-multus-certs\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143177    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-config\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143196    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-web-config\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143204    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143250    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-kubernetes\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143304    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-secret-metrics-server-tls\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143371    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-sys\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143404    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143434    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-tmp-dir\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143483    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovnkube-script-lib\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143506    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysctl-d\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143523    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/14492824-39e2-4e98-a91d-ac81382ad70f-audit-log\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143515    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-tmp-dir\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143546    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d59c905e-0042-4777-a2fe-d5d5ae04e47d-serviceca\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143585    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8dqb\" (UniqueName: \"kubernetes.io/projected/78221f6f-2223-4ac2-92da-b763ff2a6b39-kube-api-access-p8dqb\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143619    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-k8s-cni-cncf-io\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.143719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143720    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-config\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143792    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/14492824-39e2-4e98-a91d-ac81382ad70f-audit-log\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143822    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-netns\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143848    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-etc-kubernetes\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143902    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovnkube-script-lib\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143908    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b72267e-e7f7-4115-af3e-14a382e659af-metrics-tls\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143949    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.143976    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmb9\" (UniqueName: \"kubernetes.io/projected/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-kube-api-access-wvmb9\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144029    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d-agent-certs\") pod \"konnectivity-agent-26p8v\" (UID: \"83dcdc7a-f1ba-4570-bed5-a40b0d3b213d\") " pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144052    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144077    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144104    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144133    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysconfig\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144161    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144170    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b72267e-e7f7-4115-af3e-14a382e659af-metrics-tls\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144196    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144201    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d59c905e-0042-4777-a2fe-d5d5ae04e47d-serviceca\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144190    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4gb2\" (UniqueName: \"kubernetes.io/projected/025ece4b-9534-475f-b96d-0cb3edfe3ac3-kube-api-access-g4gb2\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144303    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d-agent-certs\") pod \"konnectivity-agent-26p8v\" (UID: \"83dcdc7a-f1ba-4570-bed5-a40b0d3b213d\") " pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144310    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-node-log\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144347    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-wtmp\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144377    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-data-volume\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144405    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144455    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144493    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144497    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144537    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144573    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-data-volume\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144592    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-conf-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144652    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144654    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-sys\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144692    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-textfile\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144733    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-device-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144763    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d-konnectivity-ca\") pod \"konnectivity-agent-26p8v\" (UID: \"83dcdc7a-f1ba-4570-bed5-a40b0d3b213d\") " pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144775    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-textfile\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144797    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144802    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ff9623fb-ce84-42fc-aa52-fc263d66ec90-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144924    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144944    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ff9623fb-ce84-42fc-aa52-fc263d66ec90-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144953    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83dcdc7a-f1ba-4570-bed5-a40b0d3b213d-konnectivity-ca\") pod \"konnectivity-agent-26p8v\" (UID: \"83dcdc7a-f1ba-4570-bed5-a40b0d3b213d\") " pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.144976    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145022    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145042    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-tuned\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.145033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145058    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-kubelet\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145075    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145096    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jzs\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-kube-api-access-j4jzs\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145097    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145116    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-kubelet\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145147    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145174    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5x28\" (UniqueName: \"kubernetes.io/projected/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-kube-api-access-c5x28\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145202    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-tuned\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145235    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-cni-bin\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145256    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-hostroot\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145193    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145290    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-socket-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145302    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145355    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145385    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-cni-netd\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145413    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14492824-39e2-4e98-a91d-ac81382ad70f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145440    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-system-cni-dir\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145544    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxbqz\" (UniqueName: \"kubernetes.io/projected/1827f1a3-a7e6-4d57-907a-763220ce3e0e-kube-api-access-gxbqz\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145641    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145647    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145654    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145705    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-secret-telemeter-client\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145724    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-ovn\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145740    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14492824-39e2-4e98-a91d-ac81382ad70f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145753    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovnkube-config\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145783    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145828    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv5cq\" (UniqueName: \"kubernetes.io/projected/0162c6de-30e1-4e0b-949e-43b813b1e953-kube-api-access-pv5cq\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145852    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-serving-certs-ca-bundle\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145877    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145907    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-run\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145931    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-cni-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145959    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c0067039-d47d-4ee4-8e02-1e24cdc26732-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5d9mm\" (UID: \"c0067039-d47d-4ee4-8e02-1e24cdc26732\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.145982    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovnkube-config\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146037    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-secret-telemeter-client\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146055    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146085    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-crio-socket\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146110    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-tls\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146144    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146167    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysctl-conf\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146191    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cnibin\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146215    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146233    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146254    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146314    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146313    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c0067039-d47d-4ee4-8e02-1e24cdc26732-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-5d9mm\" (UID: \"c0067039-d47d-4ee4-8e02-1e24cdc26732\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146398    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-serving-certs-ca-bundle\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146240    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9vf\" (UniqueName: \"kubernetes.io/projected/0cf6192d-d2e9-498b-b582-32305e1572d7-kube-api-access-qh9vf\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146439    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146476    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146496    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146524    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76ddl\" (UniqueName: \"kubernetes.io/projected/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-kube-api-access-76ddl\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146553    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-telemeter-client-tls\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146577    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146583    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-tls\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146602    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-host\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146633    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/14492824-39e2-4e98-a91d-ac81382ad70f-metrics-server-audit-profiles\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146657    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146683    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146711    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovn-node-metrics-cert\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146712    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146737    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-socket-dir-parent\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146768    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146794    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9623fb-ce84-42fc-aa52-fc263d66ec90-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146820    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146844    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-slash\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146867    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-run-netns\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146793    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-telemeter-client-tls\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146894    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-secret-metrics-server-client-certs\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.146983    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/14492824-39e2-4e98-a91d-ac81382ad70f-metrics-server-audit-profiles\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147012    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-ovn-node-metrics-cert\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147055    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csd2s\" (UniqueName: \"kubernetes.io/projected/9b72267e-e7f7-4115-af3e-14a382e659af-kube-api-access-csd2s\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147086    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-metrics-client-ca\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147106    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/14492824-39e2-4e98-a91d-ac81382ad70f-secret-metrics-server-client-certs\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147113    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-systemd\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147142    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfhm\" (UniqueName: \"kubernetes.io/projected/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-kube-api-access-jjfhm\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147169    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/259d31ef-0ef7-4655-9121-88fd07468c38-kubelet-config\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147190    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.147158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147197    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmz9r\" (UniqueName: \"kubernetes.io/projected/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-api-access-jmz9r\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147230    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147253    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147277    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff9623fb-ce84-42fc-aa52-fc263d66ec90-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147334    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147444    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/025ece4b-9534-475f-b96d-0cb3edfe3ac3-metrics-client-ca\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147464    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147489    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147488    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147518    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-var-lib-kubelet\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147545    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147573    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-iptables-alerter-script\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147630    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-metrics-client-ca\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147665    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147759    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-iptables-alerter-script\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147842    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147887    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147901    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147926    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8bmf\" (UniqueName: \"kubernetes.io/projected/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-kube-api-access-p8bmf\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147943    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-host-slash\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.147961    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148015    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148034    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148062    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d59c905e-0042-4777-a2fe-d5d5ae04e47d-host\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148080    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-var-lib-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148080    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148096    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148102    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148175    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148222    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-lib-modules\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148242    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9b72267e-e7f7-4115-af3e-14a382e659af-config-volume\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148250    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148267    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b72267e-e7f7-4115-af3e-14a382e659af-tmp-dir\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148271    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/025ece4b-9534-475f-b96d-0cb3edfe3ac3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148286    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx6p7\" (UniqueName: \"kubernetes.io/projected/d9215ec4-711e-400c-852c-aaaebbd7a7ec-kube-api-access-hx6p7\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148297    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78221f6f-2223-4ac2-92da-b763ff2a6b39-metrics-certs\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148385    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9b72267e-e7f7-4115-af3e-14a382e659af-config-volume\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148385    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148415    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-cni-bin\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148425    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-metrics-client-ca\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148441    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-cnibin\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148483    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b72267e-e7f7-4115-af3e-14a382e659af-tmp-dir\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:35.150800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.148623    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.158648 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.158621    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:35.164225 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.164196    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tkl\" (UniqueName: \"kubernetes.io/projected/c0067039-d47d-4ee4-8e02-1e24cdc26732-kube-api-access-b6tkl\") pod \"monitoring-plugin-77f4f76594-5d9mm\" (UID: \"c0067039-d47d-4ee4-8e02-1e24cdc26732\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:35.164396 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.164281    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgpqc\" (UniqueName: \"kubernetes.io/projected/d59c905e-0042-4777-a2fe-d5d5ae04e47d-kube-api-access-vgpqc\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:27:35.164396 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.164370    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwdjq\" (UniqueName: \"kubernetes.io/projected/18c61308-36a9-4eb4-9cc9-ac85bf1bdd60-kube-api-access-fwdjq\") pod \"network-check-target-2hzq7\" (UID: \"18c61308-36a9-4eb4-9cc9-ac85bf1bdd60\") " pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:27:35.164396 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.164386    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p26c\" (UniqueName: \"kubernetes.io/projected/4c8202fc-395e-4188-af53-19a2d40a55fe-kube-api-access-5p26c\") pod \"ingress-canary-nkq67\" (UID: \"4c8202fc-395e-4188-af53-19a2d40a55fe\") " pod="openshift-ingress-canary/ingress-canary-nkq67"
Nov 02 12:27:35.164519 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.164385    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw7dg\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-kube-api-access-lw7dg\") pod \"alertmanager-main-0\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:35.164738 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.164721    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnwn9\" (UniqueName: \"kubernetes.io/projected/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-kube-api-access-xnwn9\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:35.165083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.165067    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts8bt\" (UniqueName: \"kubernetes.io/projected/14492824-39e2-4e98-a91d-ac81382ad70f-kube-api-access-ts8bt\") pod \"metrics-server-f7db9d655-n8z5r\" (UID: \"14492824-39e2-4e98-a91d-ac81382ad70f\") " pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:35.166593 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.166574    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8dqb\" (UniqueName: \"kubernetes.io/projected/78221f6f-2223-4ac2-92da-b763ff2a6b39-kube-api-access-p8dqb\") pod \"network-metrics-daemon-2vkgz\" (UID: \"78221f6f-2223-4ac2-92da-b763ff2a6b39\") " pod="openshift-multus/network-metrics-daemon-2vkgz"
Nov 02 12:27:35.167327 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.167308    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97wkz\" (UniqueName: \"kubernetes.io/projected/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-kube-api-access-97wkz\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.170486 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.170454    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmb9\" (UniqueName: \"kubernetes.io/projected/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-kube-api-access-wvmb9\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:27:35.170973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.170941    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5x28\" (UniqueName: \"kubernetes.io/projected/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-kube-api-access-c5x28\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:27:35.171260 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.171240    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxbqz\" (UniqueName: \"kubernetes.io/projected/1827f1a3-a7e6-4d57-907a-763220ce3e0e-kube-api-access-gxbqz\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.171260 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.171248    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jzs\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-kube-api-access-j4jzs\") pod \"prometheus-k8s-0\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:35.171412 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.171384    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx6p7\" (UniqueName: \"kubernetes.io/projected/d9215ec4-711e-400c-852c-aaaebbd7a7ec-kube-api-access-hx6p7\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.171729 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.171695    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76ddl\" (UniqueName: \"kubernetes.io/projected/3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72-kube-api-access-76ddl\") pod \"thanos-querier-57f985446f-hkswt\" (UID: \"3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72\") " pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:35.171798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.171735    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv5cq\" (UniqueName: \"kubernetes.io/projected/0162c6de-30e1-4e0b-949e-43b813b1e953-kube-api-access-pv5cq\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.171798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.171739    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4gb2\" (UniqueName: \"kubernetes.io/projected/025ece4b-9534-475f-b96d-0cb3edfe3ac3-kube-api-access-g4gb2\") pod \"telemeter-client-cf7f64998-5wcsl\" (UID: \"025ece4b-9534-475f-b96d-0cb3edfe3ac3\") " pod="openshift-monitoring/telemeter-client-cf7f64998-5wcsl"
Nov 02 12:27:35.172322 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.172302    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9vf\" (UniqueName: \"kubernetes.io/projected/0cf6192d-d2e9-498b-b582-32305e1572d7-kube-api-access-qh9vf\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.172389 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.172372    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csd2s\" (UniqueName: \"kubernetes.io/projected/9b72267e-e7f7-4115-af3e-14a382e659af-kube-api-access-csd2s\") pod \"dns-default-snnjx\" (UID: \"9b72267e-e7f7-4115-af3e-14a382e659af\") " pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:35.172491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.172473    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmz9r\" (UniqueName: \"kubernetes.io/projected/ff9623fb-ce84-42fc-aa52-fc263d66ec90-kube-api-access-jmz9r\") pod \"kube-state-metrics-5dc8d7d89b-v57wr\" (UID: \"ff9623fb-ce84-42fc-aa52-fc263d66ec90\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-v57wr"
Nov 02 12:27:35.174562 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.174537    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfhm\" (UniqueName: \"kubernetes.io/projected/5bb243e5-8e99-4391-9a55-6e3a8f82ae0c-kube-api-access-jjfhm\") pod \"openshift-state-metrics-7bd7489957-x9flx\" (UID: \"5bb243e5-8e99-4391-9a55-6e3a8f82ae0c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-x9flx"
Nov 02 12:27:35.176364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.176339    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8bmf\" (UniqueName: \"kubernetes.io/projected/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-kube-api-access-p8bmf\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.249371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249285    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysctl-d\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.249371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249322    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-k8s-cni-cncf-io\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249337    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-netns\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249351    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-etc-kubernetes\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249369    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysconfig\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249388    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-node-log\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249407    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-wtmp\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249414    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-k8s-cni-cncf-io\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249432    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249428    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-netns\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249439    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-etc-kubernetes\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249459    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-node-log\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249448    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-conf-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249474    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-conf-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249502    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249501    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysctl-d\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249511    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-sys\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249540    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-device-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249545    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-sys\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249560    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-kubelet\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249564    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-node-exporter-wtmp\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249585    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-kubelet\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249593    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysconfig\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249601    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249597    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-device-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249610    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-kubelet\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249619    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-cni-bin\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249627    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-kubelet\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249644    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249673    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-hostroot\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249690    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-socket-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249726    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-hostroot\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249725    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-cni-bin\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249744    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-cni-netd\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249749    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-socket-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249763    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-system-cni-dir\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249779    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.249782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249795    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-ovn\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249814    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-run\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249829    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-system-cni-dir\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249830    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-cni-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249863    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249869    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-ovn\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249881    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-crio-socket\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249878    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-cni-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249890    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-run\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249907    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysctl-conf\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249918    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a-crio-socket\") pod \"insights-runtime-extractor-8mqtc\" (UID: \"7f93fec2-d1e5-4efc-82c6-b029ce9e3d7a\") " pod="openshift-insights/insights-runtime-extractor-8mqtc"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249923    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cnibin\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249941    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-host\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249963    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249972    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1827f1a3-a7e6-4d57-907a-763220ce3e0e-cnibin\") pod \"multus-additional-cni-plugins-fspl8\" (UID: \"1827f1a3-a7e6-4d57-907a-763220ce3e0e\") " pod="openshift-multus/multus-additional-cni-plugins-fspl8"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249980    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-socket-dir-parent\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249987    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-host\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.249878    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-cni-netd\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250025    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-slash\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250040    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-multus-socket-dir-parent\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250048    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-run-netns\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250067    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250068    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-sysctl-conf\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250083    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-systemd\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250094    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-slash\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250109    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/259d31ef-0ef7-4655-9121-88fd07468c38-kubelet-config\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250132    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-var-lib-kubelet\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250078    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-run-netns\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250159    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-host-slash\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250187    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-var-lib-kubelet\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250191    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d59c905e-0042-4777-a2fe-d5d5ae04e47d-host\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250197    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/259d31ef-0ef7-4655-9121-88fd07468c38-kubelet-config\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250160    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-systemd\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250218    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d59c905e-0042-4777-a2fe-d5d5ae04e47d-host\") pod \"node-ca-mdzbn\" (UID: \"d59c905e-0042-4777-a2fe-d5d5ae04e47d\") " pod="openshift-image-registry/node-ca-mdzbn"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250216    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-var-lib-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250244    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-var-lib-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250244    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250271    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-run-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250271    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-lib-modules\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250193    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a41b8e21-8e18-4e58-85d9-c24ba888b2f6-host-slash\") pod \"iptables-alerter-c4mfn\" (UID: \"a41b8e21-8e18-4e58-85d9-c24ba888b2f6\") " pod="openshift-network-operator/iptables-alerter-c4mfn"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250297    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-cni-bin\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250314    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-cnibin\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250334    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/259d31ef-0ef7-4655-9121-88fd07468c38-dbus\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250345    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-lib-modules\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250356    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-cni-bin\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250360    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-etc-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250385    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-cnibin\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250394    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/259d31ef-0ef7-4655-9121-88fd07468c38-dbus\") pod \"global-pull-secret-syncer-x7jpb\" (UID: \"259d31ef-0ef7-4655-9121-88fd07468c38\") " pod="kube-system/global-pull-secret-syncer-x7jpb"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250392    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-etc-openvswitch\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250400    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-modprobe-d\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250451    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-system-cni-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250468    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-modprobe-d\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250475    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-root\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250493    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-hosts-file\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250511    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-systemd-units\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250531    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-systemd\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250542    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d9215ec4-711e-400c-852c-aaaebbd7a7ec-root\") pod \"node-exporter-lbm4n\" (UID: \"d9215ec4-711e-400c-852c-aaaebbd7a7ec\") " pod="openshift-monitoring/node-exporter-lbm4n"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250556    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-registration-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250566    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-system-cni-dir\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250568    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b89c2be7-3500-4d9a-ac82-4ef367b83e3e-hosts-file\") pod \"node-resolver-k49jr\" (UID: \"b89c2be7-3500-4d9a-ac82-4ef367b83e3e\") " pod="openshift-dns/node-resolver-k49jr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250582    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-systemd-units\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250590    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-systemd\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250593    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-sys-fs\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250613    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-registration-dir\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250626    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-cni-multus\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250635    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0cf6192d-d2e9-498b-b582-32305e1572d7-sys-fs\") pod \"aws-ebs-csi-driver-node-p4pql\" (UID: \"0cf6192d-d2e9-498b-b582-32305e1572d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p4pql"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250644    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-log-socket\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250662    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-log-socket\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250670    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-multus-certs\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250688    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250684    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-var-lib-cni-multus\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250702    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-kubernetes\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250722    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0162c6de-30e1-4e0b-949e-43b813b1e953-host-run-multus-certs\") pod \"multus-5qqx4\" (UID: \"0162c6de-30e1-4e0b-949e-43b813b1e953\") " pod="openshift-multus/multus-5qqx4"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250731    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-sys\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250744    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471-host-run-ovn-kubernetes\") pod \"ovnkube-node-l9dwk\" (UID: \"6f14c2fa-a8fc-4569-8aaa-5be8ea7ab471\") " pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250761    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-etc-kubernetes\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:35.250926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:35.250815    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/718bc5cd-de70-4920-9e8f-5bc14fe1f45c-sys\") pod \"tuned-9w9rr\" (UID: \"718bc5cd-de70-4920-9e8f-5bc14fe1f45c\") " pod="openshift-cluster-node-tuning-operator/tuned-9w9rr"
Nov 02 12:27:36.453422 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:36.453392    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f7db9d655-n8z5r"
Nov 02 12:27:36.573116 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:36.573091    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:36.972541 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:36.972508    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-snnjx"
Nov 02 12:27:38.339015 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:38.338968    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57f985446f-hkswt"
Nov 02 12:27:39.402196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:39.402164    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:41.555278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:41.555238    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:41.678244 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:41.678211    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:41.818198 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:41.818124    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:41.934508 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:41.934480    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:42.948879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:42.948839    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:27:43.007950 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:43.007912    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-5d9mm"
Nov 02 12:27:44.014657 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:44.014591    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:44.015017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:44.014767    5879 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 12:27:44.819785 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:44.819749    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:27:45.617825 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:45.617787    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:46.142777 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:46.142743    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:46.265432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:46.265394    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:46.492558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:46.492461    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:46.607520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:46.607490    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:46.843050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:46.842945    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 12:27:46.843504 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:46.843422    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="alertmanager" containerID="cri-o://dcadc026789767fa11102be451e08f929a46592db1cc93397533ff87277d072c" gracePeriod=120
Nov 02 12:27:46.843504 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:46.843433    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="kube-rbac-proxy-web" containerID="cri-o://71348250e126d8505c1fcee0d51e377d401696bdcfb4f9d9142ed67c75484ea0" gracePeriod=120
Nov 02 12:27:46.843504 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:46.843465    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="config-reloader" containerID="cri-o://bdb53a466aceccc533a9cde054805fccbcbf84f294acee9b78014508a5535fc4" gracePeriod=120
Nov 02 12:27:46.843504 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:46.843429    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="kube-rbac-proxy-metric" containerID="cri-o://4e4bd02e09d614bad4b3122c985d25b55035e77442d8ec2bd69aee266edc7a9e" gracePeriod=120
Nov 02 12:27:46.843696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:46.843467    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="prom-label-proxy" containerID="cri-o://0073d1967d0b898124408fc1da498d48cba29fd20381c08d67433164d100f182" gracePeriod=120
Nov 02 12:27:46.843696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:46.843475    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="kube-rbac-proxy" containerID="cri-o://bb0a83fe82f634b603d022df5d15a9ac84487b419c85adfa4d471a95896c1767" gracePeriod=120
Nov 02 12:27:47.136369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:47.136338    5879 generic.go:356] "Generic (PLEG): container finished" podID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerID="0073d1967d0b898124408fc1da498d48cba29fd20381c08d67433164d100f182" exitCode=0
Nov 02 12:27:47.136369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:47.136362    5879 generic.go:356] "Generic (PLEG): container finished" podID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerID="4e4bd02e09d614bad4b3122c985d25b55035e77442d8ec2bd69aee266edc7a9e" exitCode=0
Nov 02 12:27:47.136369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:47.136370    5879 generic.go:356] "Generic (PLEG): container finished" podID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerID="bb0a83fe82f634b603d022df5d15a9ac84487b419c85adfa4d471a95896c1767" exitCode=0
Nov 02 12:27:47.136369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:47.136377    5879 generic.go:356] "Generic (PLEG): container finished" podID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerID="bdb53a466aceccc533a9cde054805fccbcbf84f294acee9b78014508a5535fc4" exitCode=0
Nov 02 12:27:47.136635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:47.136384    5879 generic.go:356] "Generic (PLEG): container finished" podID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerID="dcadc026789767fa11102be451e08f929a46592db1cc93397533ff87277d072c" exitCode=0
Nov 02 12:27:47.136635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:47.136413    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerDied","Data":"0073d1967d0b898124408fc1da498d48cba29fd20381c08d67433164d100f182"}
Nov 02 12:27:47.136635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:47.136500    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerDied","Data":"4e4bd02e09d614bad4b3122c985d25b55035e77442d8ec2bd69aee266edc7a9e"}
Nov 02 12:27:47.136635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:47.136529    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerDied","Data":"bb0a83fe82f634b603d022df5d15a9ac84487b419c85adfa4d471a95896c1767"}
Nov 02 12:27:47.136635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:47.136556    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerDied","Data":"bdb53a466aceccc533a9cde054805fccbcbf84f294acee9b78014508a5535fc4"}
Nov 02 12:27:47.136635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:47.136583    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerDied","Data":"dcadc026789767fa11102be451e08f929a46592db1cc93397533ff87277d072c"}
Nov 02 12:27:48.144220 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.144189    5879 generic.go:356] "Generic (PLEG): container finished" podID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerID="71348250e126d8505c1fcee0d51e377d401696bdcfb4f9d9142ed67c75484ea0" exitCode=0
Nov 02 12:27:48.144625 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.144230    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerDied","Data":"71348250e126d8505c1fcee0d51e377d401696bdcfb4f9d9142ed67c75484ea0"}
Nov 02 12:27:48.621263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.621233    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:48.665126 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665104    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-main-tls\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.665270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665170    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.665270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665202    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-trusted-ca-bundle\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.665270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665224    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.665270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665249    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-metrics-client-ca\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.665270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665270    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-web-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-web-config\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.665542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665293    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-main-db\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.665542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665332    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-out\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.665542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665364    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-tls-assets\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-tls-assets\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.665542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665401    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-volume\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-volume\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.665542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665436    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-kube-api-access-lw7dg\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-kube-api-access-lw7dg\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.665812 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665643    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-trusted-ca-bundle" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:48.665812 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.665720    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-metrics-client-ca" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:48.666429 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.666077    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-main-db" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:48.668411 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.668383    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-metric" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:48.668895 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.668872    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-main-tls" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:48.669152 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.669127    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-out" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:48.669152 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.669125    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-tls-assets" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:48.669291 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.669147    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-web" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:48.669291 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.669243    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-volume" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:48.669291 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.669245    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-kube-api-access-lw7dg" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "kube-api-access-lw7dg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:48.679818 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.679796    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-web-config" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:48.766236 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766136    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.766236 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766186    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-cluster-tls-config\") pod \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\" (UID: \"70e92edc-76ab-4aac-9928-4fd21b20b7e8\") "
Nov 02 12:27:48.766465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766320    5879 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-out\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:48.766465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766330    5879 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-tls-assets\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:48.766465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766340    5879 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:48.766465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766353    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw7dg\" (UniqueName: \"kubernetes.io/projected/70e92edc-76ab-4aac-9928-4fd21b20b7e8-kube-api-access-lw7dg\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:48.766465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766367    5879 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-main-tls\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:48.766465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766377    5879 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:48.766465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766386    5879 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:48.766465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766397    5879 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:48.766465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766406    5879 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70e92edc-76ab-4aac-9928-4fd21b20b7e8-metrics-client-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:48.766465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766438    5879 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-web-config\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:48.766465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.766450    5879 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70e92edc-76ab-4aac-9928-4fd21b20b7e8-alertmanager-main-db\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:48.768314 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.768282    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:48.772047 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.772018    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-cluster-tls-config" pod "70e92edc-76ab-4aac-9928-4fd21b20b7e8" (UID: "70e92edc-76ab-4aac-9928-4fd21b20b7e8"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:48.867176 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.867135    5879 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-cluster-tls-config\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:48.867176 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:48.867172    5879 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70e92edc-76ab-4aac-9928-4fd21b20b7e8-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:49.151582 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.151549    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70e92edc-76ab-4aac-9928-4fd21b20b7e8","Type":"ContainerDied","Data":"b35a43d4116b66143b827d78db7d612bd73cfd3adfa90f575dab329f5fb440f6"}
Nov 02 12:27:49.151582 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.151583    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.152043 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.151627    5879 scope.go:122] "RemoveContainer" containerID="0073d1967d0b898124408fc1da498d48cba29fd20381c08d67433164d100f182"
Nov 02 12:27:49.173594 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.173159    5879 scope.go:122] "RemoveContainer" containerID="4e4bd02e09d614bad4b3122c985d25b55035e77442d8ec2bd69aee266edc7a9e"
Nov 02 12:27:49.181445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.181423    5879 scope.go:122] "RemoveContainer" containerID="bb0a83fe82f634b603d022df5d15a9ac84487b419c85adfa4d471a95896c1767"
Nov 02 12:27:49.188923 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.188904    5879 scope.go:122] "RemoveContainer" containerID="71348250e126d8505c1fcee0d51e377d401696bdcfb4f9d9142ed67c75484ea0"
Nov 02 12:27:49.196104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.196085    5879 scope.go:122] "RemoveContainer" containerID="bdb53a466aceccc533a9cde054805fccbcbf84f294acee9b78014508a5535fc4"
Nov 02 12:27:49.203612 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.203593    5879 scope.go:122] "RemoveContainer" containerID="dcadc026789767fa11102be451e08f929a46592db1cc93397533ff87277d072c"
Nov 02 12:27:49.211359 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.210217    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 12:27:49.214685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.214661    5879 scope.go:122] "RemoveContainer" containerID="8cc3c8ee3ebb3173b9b413353bd7104ed4d37655ff2e43588b51ebc2df7b728c"
Nov 02 12:27:49.223795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.223770    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 12:27:49.247916 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.247892    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 12:27:49.248310 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248287    5879 cpu_manager.go:414] "containerMap: removing container" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="kube-rbac-proxy-metric"
Nov 02 12:27:49.248310 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248307    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="kube-rbac-proxy-metric"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248319    5879 cpu_manager.go:414] "containerMap: removing container" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="alertmanager"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248324    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="alertmanager"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248338    5879 cpu_manager.go:414] "containerMap: removing container" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="kube-rbac-proxy-web"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248345    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="kube-rbac-proxy-web"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248354    5879 cpu_manager.go:414] "containerMap: removing container" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="init-config-reloader"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248361    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="init-config-reloader"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248380    5879 cpu_manager.go:414] "containerMap: removing container" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="prom-label-proxy"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248385    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="prom-label-proxy"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248403    5879 cpu_manager.go:414] "containerMap: removing container" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="kube-rbac-proxy"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248407    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="kube-rbac-proxy"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248415    5879 cpu_manager.go:414] "containerMap: removing container" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="config-reloader"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248419    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="config-reloader"
Nov 02 12:27:49.248480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248480    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="prom-label-proxy"
Nov 02 12:27:49.248876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248491    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="kube-rbac-proxy"
Nov 02 12:27:49.248876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248496    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="config-reloader"
Nov 02 12:27:49.248876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248502    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="kube-rbac-proxy-metric"
Nov 02 12:27:49.248876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248507    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="alertmanager"
Nov 02 12:27:49.248876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248515    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="kube-rbac-proxy-web"
Nov 02 12:27:49.248876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.248519    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" containerName="init-config-reloader"
Nov 02 12:27:49.259914 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.259880    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.270777 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.270752    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.270869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.270789    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttctv\" (UniqueName: \"kubernetes.io/projected/ff446abc-cc59-403b-bc19-3b1d514b19a3-kube-api-access-ttctv\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.270924 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.270867    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.270960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.270925    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.270960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.270956    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff446abc-cc59-403b-bc19-3b1d514b19a3-config-out\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.271059 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.270984    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.271059 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.271019    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff446abc-cc59-403b-bc19-3b1d514b19a3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.271131 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.271068    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.271131 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.271084    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-web-config\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.271131 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.271099    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff446abc-cc59-403b-bc19-3b1d514b19a3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.271131 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.271113    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff446abc-cc59-403b-bc19-3b1d514b19a3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.271290 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.271237    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff446abc-cc59-403b-bc19-3b1d514b19a3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.271290 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.271265    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.282234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.282211    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 02 12:27:49.282364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.282236    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 02 12:27:49.282364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.282333    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 02 12:27:49.282364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.282354    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 02 12:27:49.282530 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.282369    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 02 12:27:49.282530 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.282441    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 02 12:27:49.296133 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.296108    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 02 12:27:49.296419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.296401    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 02 12:27:49.296496 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.296430    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 02 12:27:49.296496 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.296445    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-g5vwz\""
Nov 02 12:27:49.297262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.297242    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 12:27:49.372210 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372178    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372210 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372218    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff446abc-cc59-403b-bc19-3b1d514b19a3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372283    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372312    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-web-config\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372331    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff446abc-cc59-403b-bc19-3b1d514b19a3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372347    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff446abc-cc59-403b-bc19-3b1d514b19a3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372387    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff446abc-cc59-403b-bc19-3b1d514b19a3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372426    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372475    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372505    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttctv\" (UniqueName: \"kubernetes.io/projected/ff446abc-cc59-403b-bc19-3b1d514b19a3-kube-api-access-ttctv\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372553    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372587    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372624    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff446abc-cc59-403b-bc19-3b1d514b19a3-config-out\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.372814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372805    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff446abc-cc59-403b-bc19-3b1d514b19a3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.373088 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.372816    5879 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 12:27:49.373524 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.373447    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff446abc-cc59-403b-bc19-3b1d514b19a3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.373696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.373671    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff446abc-cc59-403b-bc19-3b1d514b19a3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.375936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.375899    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-web-config\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.376234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.376125    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.376332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.376242    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff446abc-cc59-403b-bc19-3b1d514b19a3-config-out\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.376462 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.376445    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.376555 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.376538    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff446abc-cc59-403b-bc19-3b1d514b19a3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.376759 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.376724    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.376875 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.376820    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.376939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.376888    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.377793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.377768    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff446abc-cc59-403b-bc19-3b1d514b19a3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.397587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.397561    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttctv\" (UniqueName: \"kubernetes.io/projected/ff446abc-cc59-403b-bc19-3b1d514b19a3-kube-api-access-ttctv\") pod \"alertmanager-main-0\" (UID: \"ff446abc-cc59-403b-bc19-3b1d514b19a3\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.636754 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.636722    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 02 12:27:49.920510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.920426    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:49.999300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:49.999278    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 02 12:27:50.001368 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:50.001342    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff446abc_cc59_403b_bc19_3b1d514b19a3.slice/crio-31b0bb116d12c310621917e31e821e1e50bd49361ad0a28d830117931e8478c2 WatchSource:0}: Error finding container 31b0bb116d12c310621917e31e821e1e50bd49361ad0a28d830117931e8478c2: Status 404 returned error can't find the container with id 31b0bb116d12c310621917e31e821e1e50bd49361ad0a28d830117931e8478c2
Nov 02 12:27:50.015646 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:50.015613    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e92edc-76ab-4aac-9928-4fd21b20b7e8" path="/var/lib/kubelet/pods/70e92edc-76ab-4aac-9928-4fd21b20b7e8/volumes"
Nov 02 12:27:50.054944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:50.054916    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:50.156633 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:50.156600    5879 generic.go:356] "Generic (PLEG): container finished" podID="ff446abc-cc59-403b-bc19-3b1d514b19a3" containerID="0e53693112bc8164ece4a048b0a5a4bee9402f4c5a2269abc8fe641889a04e95" exitCode=0
Nov 02 12:27:50.157084 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:50.156696    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff446abc-cc59-403b-bc19-3b1d514b19a3","Type":"ContainerDied","Data":"0e53693112bc8164ece4a048b0a5a4bee9402f4c5a2269abc8fe641889a04e95"}
Nov 02 12:27:50.157084 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:50.156798    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff446abc-cc59-403b-bc19-3b1d514b19a3","Type":"ContainerStarted","Data":"31b0bb116d12c310621917e31e821e1e50bd49361ad0a28d830117931e8478c2"}
Nov 02 12:27:50.712425 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:50.712334    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-l9dwk"
Nov 02 12:27:51.166769 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.166738    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff446abc-cc59-403b-bc19-3b1d514b19a3","Type":"ContainerStarted","Data":"ca56dd1c5e68207aeafc86b52a1c6a8df56e21748ff6464958394a7735497717"}
Nov 02 12:27:51.167175 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.166801    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff446abc-cc59-403b-bc19-3b1d514b19a3","Type":"ContainerStarted","Data":"f84e9a7dfecffaeab1fed3971fb376a4b33720bed115314f9adb55a4227c5578"}
Nov 02 12:27:51.167175 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.166828    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff446abc-cc59-403b-bc19-3b1d514b19a3","Type":"ContainerStarted","Data":"af0157f2bafaef9c510f783d66bf878503cedd9577039ce720d3f648e44a79ff"}
Nov 02 12:27:51.167175 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.166853    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff446abc-cc59-403b-bc19-3b1d514b19a3","Type":"ContainerStarted","Data":"64f011f47a449313e531eed3cd8d34df3a06e9952e8619805534d4c6634e0b20"}
Nov 02 12:27:51.167175 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.166878    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff446abc-cc59-403b-bc19-3b1d514b19a3","Type":"ContainerStarted","Data":"b13aad2a1fbe34034568189484dd99554d6aaba836f9238a37e199238f29d17b"}
Nov 02 12:27:51.167175 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.166901    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff446abc-cc59-403b-bc19-3b1d514b19a3","Type":"ContainerStarted","Data":"9cc4292baaf2f2b21f359a8a6f90a63689a3d079545effde8a35741e523b5da9"}
Nov 02 12:27:51.224069 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.224019    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.209071981 podStartE2EDuration="2.223986949s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.914968ms" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:51.223774535 +0000 UTC m=+17.330477399" watchObservedRunningTime="2026-11-02 12:27:51.223986949 +0000 UTC m=+17.330689825"
Nov 02 12:27:51.472618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.472538    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 12:27:51.472890 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.472850    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="kube-rbac-proxy" containerID="cri-o://b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22" gracePeriod=600
Nov 02 12:27:51.473065 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.472883    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="kube-rbac-proxy-web" containerID="cri-o://81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05" gracePeriod=600
Nov 02 12:27:51.473065 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.472904    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="config-reloader" containerID="cri-o://6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30" gracePeriod=600
Nov 02 12:27:51.473065 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.472844    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="prometheus" containerID="cri-o://64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400" gracePeriod=600
Nov 02 12:27:51.473065 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.472883    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="thanos-sidecar" containerID="cri-o://efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83" gracePeriod=600
Nov 02 12:27:51.473065 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.472920    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="kube-rbac-proxy-thanos" containerID="cri-o://05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224" gracePeriod=600
Nov 02 12:27:51.721872 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.721848    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:51.793950 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.793868    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-metrics-client-certs\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.793950 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.793921    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-thanos-prometheus-http-client-file\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794197 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.793957    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794197 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794075    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-serving-certs-ca-bundle\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794197 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794122    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-1\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794197 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794150    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-0\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794197 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794176    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-metrics-client-ca\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794200    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-tls\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794218    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-grpc-tls\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794241    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-kubelet-serving-ca-bundle\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794274    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-trusted-ca-bundle\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794328    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-db\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794385    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794434    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-config\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794472    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-config-out\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794512    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-tls-assets\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-tls-assets\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794562    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-2\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794598    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-web-config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-web-config\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794627    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-kube-rbac-proxy\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.794767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.794686    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-kube-api-access-j4jzs\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-kube-api-access-j4jzs\") pod \"ffb2d74e-5ea9-48e3-98b8-53963e608786\" (UID: \"ffb2d74e-5ea9-48e3-98b8-53963e608786\") "
Nov 02 12:27:51.795091 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.795060    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-serving-certs-ca-bundle" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:51.795536 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.795502    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-1" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:51.796014 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.795962    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-metrics-client-ca" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:51.796379 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.796353    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-trusted-ca-bundle" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:51.797221 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.797193    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-metrics-client-certs" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:51.797337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.797273    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-thanos-prometheus-http-client-file" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:51.797617 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.797589    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-kubelet-serving-ca-bundle" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:51.797800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.797771    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-2" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:51.798257 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.798231    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-db" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:51.799214 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.799186    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-kube-api-access-j4jzs" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "kube-api-access-j4jzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:51.799318 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.799288    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-0" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:27:51.799674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.799644    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-kube-rbac-proxy-web" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:51.799781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.799748    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-grpc-tls" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:51.799838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.799812    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-config" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:51.800041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.799970    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-tls" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:51.803310 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.803241    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-tls-assets" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:27:51.803310 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.803273    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-thanos-sidecar-tls" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:51.803310 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.803284    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-config-out" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:27:51.804455 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.804430    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-kube-rbac-proxy" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:51.812139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.812106    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-web-config" pod "ffb2d74e-5ea9-48e3-98b8-53963e608786" (UID: "ffb2d74e-5ea9-48e3-98b8-53963e608786"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:27:51.895744 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895702    5879 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-config\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.895744 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895736    5879 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-config-out\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.895744 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895751    5879 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-tls-assets\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895766    5879 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895779    5879 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-web-config\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895792    5879 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-kube-rbac-proxy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895803    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4jzs\" (UniqueName: \"kubernetes.io/projected/ffb2d74e-5ea9-48e3-98b8-53963e608786-kube-api-access-j4jzs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895816    5879 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-metrics-client-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895829    5879 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895842    5879 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895855    5879 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895868    5879 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895881    5879 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895903    5879 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-metrics-client-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895915    5879 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895928    5879 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-grpc-tls\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895941    5879 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895956    5879 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895970    5879 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffb2d74e-5ea9-48e3-98b8-53963e608786-prometheus-k8s-db\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:51.896058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:51.895984    5879 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffb2d74e-5ea9-48e3-98b8-53963e608786-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:27:52.173798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.173760    5879 generic.go:356] "Generic (PLEG): container finished" podID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerID="05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224" exitCode=0
Nov 02 12:27:52.173798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.173789    5879 generic.go:356] "Generic (PLEG): container finished" podID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerID="b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22" exitCode=0
Nov 02 12:27:52.173798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.173796    5879 generic.go:356] "Generic (PLEG): container finished" podID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerID="81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05" exitCode=0
Nov 02 12:27:52.173798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.173801    5879 generic.go:356] "Generic (PLEG): container finished" podID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerID="efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83" exitCode=0
Nov 02 12:27:52.173798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.173806    5879 generic.go:356] "Generic (PLEG): container finished" podID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerID="6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30" exitCode=0
Nov 02 12:27:52.174439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.173811    5879 generic.go:356] "Generic (PLEG): container finished" podID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerID="64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400" exitCode=0
Nov 02 12:27:52.174439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.173831    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerDied","Data":"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"}
Nov 02 12:27:52.174439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.173858    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.174439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.173927    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerDied","Data":"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"}
Nov 02 12:27:52.174439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.174005    5879 scope.go:122] "RemoveContainer" containerID="05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"
Nov 02 12:27:52.174439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.173988    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerDied","Data":"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"}
Nov 02 12:27:52.174439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.174058    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerDied","Data":"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"}
Nov 02 12:27:52.174439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.174101    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerDied","Data":"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"}
Nov 02 12:27:52.174439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.174140    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerDied","Data":"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"}
Nov 02 12:27:52.174439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.174182    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb2d74e-5ea9-48e3-98b8-53963e608786","Type":"ContainerDied","Data":"e27ad6ea38d49de9df8fd5a312d081f44a213d984789d22dbc95b2788a87fb46"}
Nov 02 12:27:52.182393 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.182368    5879 scope.go:122] "RemoveContainer" containerID="b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"
Nov 02 12:27:52.189639 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.189622    5879 scope.go:122] "RemoveContainer" containerID="81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"
Nov 02 12:27:52.199958 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.199935    5879 scope.go:122] "RemoveContainer" containerID="efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"
Nov 02 12:27:52.207473 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.207453    5879 scope.go:122] "RemoveContainer" containerID="6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"
Nov 02 12:27:52.214976 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.214958    5879 scope.go:122] "RemoveContainer" containerID="64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"
Nov 02 12:27:52.223015 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.222970    5879 scope.go:122] "RemoveContainer" containerID="e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"
Nov 02 12:27:52.230305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.230278    5879 scope.go:122] "RemoveContainer" containerID="05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"
Nov 02 12:27:52.230618 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:52.230593    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": container with ID starting with 05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224 not found: ID does not exist" containerID="05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"
Nov 02 12:27:52.230716 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.230627    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"} err="failed to get container status \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": rpc error: code = NotFound desc = could not find container \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": container with ID starting with 05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224 not found: ID does not exist"
Nov 02 12:27:52.230716 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.230666    5879 scope.go:122] "RemoveContainer" containerID="b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"
Nov 02 12:27:52.230963 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:52.230947    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": container with ID starting with b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22 not found: ID does not exist" containerID="b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"
Nov 02 12:27:52.231025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.230971    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"} err="failed to get container status \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": rpc error: code = NotFound desc = could not find container \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": container with ID starting with b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22 not found: ID does not exist"
Nov 02 12:27:52.231025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.230986    5879 scope.go:122] "RemoveContainer" containerID="81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"
Nov 02 12:27:52.231252 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:52.231224    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": container with ID starting with 81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05 not found: ID does not exist" containerID="81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"
Nov 02 12:27:52.231326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.231260    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"} err="failed to get container status \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": rpc error: code = NotFound desc = could not find container \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": container with ID starting with 81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05 not found: ID does not exist"
Nov 02 12:27:52.231326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.231280    5879 scope.go:122] "RemoveContainer" containerID="efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"
Nov 02 12:27:52.231553 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:52.231533    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": container with ID starting with efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83 not found: ID does not exist" containerID="efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"
Nov 02 12:27:52.231619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.231561    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"} err="failed to get container status \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": rpc error: code = NotFound desc = could not find container \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": container with ID starting with efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83 not found: ID does not exist"
Nov 02 12:27:52.231619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.231578    5879 scope.go:122] "RemoveContainer" containerID="6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"
Nov 02 12:27:52.231937 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:52.231899    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": container with ID starting with 6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30 not found: ID does not exist" containerID="6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"
Nov 02 12:27:52.232031 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.231939    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"} err="failed to get container status \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": rpc error: code = NotFound desc = could not find container \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": container with ID starting with 6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30 not found: ID does not exist"
Nov 02 12:27:52.232031 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.231956    5879 scope.go:122] "RemoveContainer" containerID="64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"
Nov 02 12:27:52.232268 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:52.232234    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": container with ID starting with 64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400 not found: ID does not exist" containerID="64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"
Nov 02 12:27:52.232268 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.232260    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"} err="failed to get container status \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": rpc error: code = NotFound desc = could not find container \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": container with ID starting with 64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400 not found: ID does not exist"
Nov 02 12:27:52.232408 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.232280    5879 scope.go:122] "RemoveContainer" containerID="e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"
Nov 02 12:27:52.232516 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:27:52.232500    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": container with ID starting with e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64 not found: ID does not exist" containerID="e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"
Nov 02 12:27:52.232548 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.232520    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"} err="failed to get container status \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": rpc error: code = NotFound desc = could not find container \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": container with ID starting with e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64 not found: ID does not exist"
Nov 02 12:27:52.232548 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.232533    5879 scope.go:122] "RemoveContainer" containerID="05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"
Nov 02 12:27:52.232759 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.232738    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"} err="failed to get container status \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": rpc error: code = NotFound desc = could not find container \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": container with ID starting with 05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224 not found: ID does not exist"
Nov 02 12:27:52.232759 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.232758    5879 scope.go:122] "RemoveContainer" containerID="b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"
Nov 02 12:27:52.233108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.232936    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"} err="failed to get container status \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": rpc error: code = NotFound desc = could not find container \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": container with ID starting with b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22 not found: ID does not exist"
Nov 02 12:27:52.233108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.232948    5879 scope.go:122] "RemoveContainer" containerID="81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"
Nov 02 12:27:52.233238 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.233218    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"} err="failed to get container status \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": rpc error: code = NotFound desc = could not find container \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": container with ID starting with 81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05 not found: ID does not exist"
Nov 02 12:27:52.233284 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.233238    5879 scope.go:122] "RemoveContainer" containerID="efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"
Nov 02 12:27:52.233499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.233479    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"} err="failed to get container status \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": rpc error: code = NotFound desc = could not find container \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": container with ID starting with efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83 not found: ID does not exist"
Nov 02 12:27:52.233549 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.233508    5879 scope.go:122] "RemoveContainer" containerID="6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"
Nov 02 12:27:52.233810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.233778    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"} err="failed to get container status \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": rpc error: code = NotFound desc = could not find container \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": container with ID starting with 6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30 not found: ID does not exist"
Nov 02 12:27:52.233810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.233802    5879 scope.go:122] "RemoveContainer" containerID="64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"
Nov 02 12:27:52.234137 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.234118    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"} err="failed to get container status \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": rpc error: code = NotFound desc = could not find container \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": container with ID starting with 64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400 not found: ID does not exist"
Nov 02 12:27:52.234217 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.234138    5879 scope.go:122] "RemoveContainer" containerID="e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"
Nov 02 12:27:52.234534 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.234496    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"} err="failed to get container status \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": rpc error: code = NotFound desc = could not find container \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": container with ID starting with e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64 not found: ID does not exist"
Nov 02 12:27:52.234534 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.234526    5879 scope.go:122] "RemoveContainer" containerID="05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"
Nov 02 12:27:52.234742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.234722    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"} err="failed to get container status \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": rpc error: code = NotFound desc = could not find container \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": container with ID starting with 05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224 not found: ID does not exist"
Nov 02 12:27:52.234796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.234743    5879 scope.go:122] "RemoveContainer" containerID="b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"
Nov 02 12:27:52.234962 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.234944    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"} err="failed to get container status \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": rpc error: code = NotFound desc = could not find container \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": container with ID starting with b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22 not found: ID does not exist"
Nov 02 12:27:52.235037 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.234963    5879 scope.go:122] "RemoveContainer" containerID="81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"
Nov 02 12:27:52.235037 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.234976    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 12:27:52.235221 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.235204    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"} err="failed to get container status \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": rpc error: code = NotFound desc = could not find container \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": container with ID starting with 81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05 not found: ID does not exist"
Nov 02 12:27:52.235262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.235221    5879 scope.go:122] "RemoveContainer" containerID="efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"
Nov 02 12:27:52.235433 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.235407    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"} err="failed to get container status \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": rpc error: code = NotFound desc = could not find container \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": container with ID starting with efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83 not found: ID does not exist"
Nov 02 12:27:52.235476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.235434    5879 scope.go:122] "RemoveContainer" containerID="6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"
Nov 02 12:27:52.235672 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.235653    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"} err="failed to get container status \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": rpc error: code = NotFound desc = could not find container \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": container with ID starting with 6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30 not found: ID does not exist"
Nov 02 12:27:52.235708 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.235672    5879 scope.go:122] "RemoveContainer" containerID="64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"
Nov 02 12:27:52.235896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.235878    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"} err="failed to get container status \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": rpc error: code = NotFound desc = could not find container \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": container with ID starting with 64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400 not found: ID does not exist"
Nov 02 12:27:52.235938 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.235897    5879 scope.go:122] "RemoveContainer" containerID="e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"
Nov 02 12:27:52.236136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.236119    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"} err="failed to get container status \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": rpc error: code = NotFound desc = could not find container \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": container with ID starting with e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64 not found: ID does not exist"
Nov 02 12:27:52.236187 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.236137    5879 scope.go:122] "RemoveContainer" containerID="05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"
Nov 02 12:27:52.236353 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.236338    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"} err="failed to get container status \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": rpc error: code = NotFound desc = could not find container \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": container with ID starting with 05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224 not found: ID does not exist"
Nov 02 12:27:52.236397 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.236354    5879 scope.go:122] "RemoveContainer" containerID="b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"
Nov 02 12:27:52.236558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.236542    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"} err="failed to get container status \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": rpc error: code = NotFound desc = could not find container \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": container with ID starting with b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22 not found: ID does not exist"
Nov 02 12:27:52.236598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.236558    5879 scope.go:122] "RemoveContainer" containerID="81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"
Nov 02 12:27:52.236771 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.236749    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"} err="failed to get container status \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": rpc error: code = NotFound desc = could not find container \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": container with ID starting with 81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05 not found: ID does not exist"
Nov 02 12:27:52.236850 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.236782    5879 scope.go:122] "RemoveContainer" containerID="efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"
Nov 02 12:27:52.237050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.237026    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"} err="failed to get container status \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": rpc error: code = NotFound desc = could not find container \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": container with ID starting with efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83 not found: ID does not exist"
Nov 02 12:27:52.237050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.237048    5879 scope.go:122] "RemoveContainer" containerID="6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"
Nov 02 12:27:52.237276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.237260    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"} err="failed to get container status \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": rpc error: code = NotFound desc = could not find container \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": container with ID starting with 6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30 not found: ID does not exist"
Nov 02 12:27:52.237276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.237276    5879 scope.go:122] "RemoveContainer" containerID="64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"
Nov 02 12:27:52.237500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.237483    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"} err="failed to get container status \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": rpc error: code = NotFound desc = could not find container \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": container with ID starting with 64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400 not found: ID does not exist"
Nov 02 12:27:52.237500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.237499    5879 scope.go:122] "RemoveContainer" containerID="e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"
Nov 02 12:27:52.237711 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.237695    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"} err="failed to get container status \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": rpc error: code = NotFound desc = could not find container \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": container with ID starting with e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64 not found: ID does not exist"
Nov 02 12:27:52.237711 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.237710    5879 scope.go:122] "RemoveContainer" containerID="05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"
Nov 02 12:27:52.237919 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.237897    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"} err="failed to get container status \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": rpc error: code = NotFound desc = could not find container \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": container with ID starting with 05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224 not found: ID does not exist"
Nov 02 12:27:52.237919 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.237914    5879 scope.go:122] "RemoveContainer" containerID="b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"
Nov 02 12:27:52.238229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.238214    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"} err="failed to get container status \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": rpc error: code = NotFound desc = could not find container \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": container with ID starting with b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22 not found: ID does not exist"
Nov 02 12:27:52.238318 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.238228    5879 scope.go:122] "RemoveContainer" containerID="81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"
Nov 02 12:27:52.238427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.238412    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"} err="failed to get container status \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": rpc error: code = NotFound desc = could not find container \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": container with ID starting with 81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05 not found: ID does not exist"
Nov 02 12:27:52.238470 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.238427    5879 scope.go:122] "RemoveContainer" containerID="efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"
Nov 02 12:27:52.238624 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.238603    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"} err="failed to get container status \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": rpc error: code = NotFound desc = could not find container \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": container with ID starting with efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83 not found: ID does not exist"
Nov 02 12:27:52.238679 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.238624    5879 scope.go:122] "RemoveContainer" containerID="6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"
Nov 02 12:27:52.238813 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.238794    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"} err="failed to get container status \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": rpc error: code = NotFound desc = could not find container \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": container with ID starting with 6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30 not found: ID does not exist"
Nov 02 12:27:52.238851 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.238814    5879 scope.go:122] "RemoveContainer" containerID="64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"
Nov 02 12:27:52.239046 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.239030    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"} err="failed to get container status \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": rpc error: code = NotFound desc = could not find container \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": container with ID starting with 64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400 not found: ID does not exist"
Nov 02 12:27:52.239099 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.239047    5879 scope.go:122] "RemoveContainer" containerID="e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"
Nov 02 12:27:52.239259 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.239243    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"} err="failed to get container status \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": rpc error: code = NotFound desc = could not find container \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": container with ID starting with e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64 not found: ID does not exist"
Nov 02 12:27:52.239297 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.239260    5879 scope.go:122] "RemoveContainer" containerID="05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"
Nov 02 12:27:52.239465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.239449    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224"} err="failed to get container status \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": rpc error: code = NotFound desc = could not find container \"05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224\": container with ID starting with 05c4e8db7b0439faa1730946b45bb62c0c18c20d5327eeb8c5cf328409688224 not found: ID does not exist"
Nov 02 12:27:52.239520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.239465    5879 scope.go:122] "RemoveContainer" containerID="b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"
Nov 02 12:27:52.239682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.239667    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22"} err="failed to get container status \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": rpc error: code = NotFound desc = could not find container \"b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22\": container with ID starting with b05f1d27b94f452fa0d11f6a6ba7b73f94665c331ea7e67d1efc257f1b665b22 not found: ID does not exist"
Nov 02 12:27:52.239727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.239682    5879 scope.go:122] "RemoveContainer" containerID="81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"
Nov 02 12:27:52.239901 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.239885    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05"} err="failed to get container status \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": rpc error: code = NotFound desc = could not find container \"81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05\": container with ID starting with 81696954384d466cb618ed71fff3a3a1f5323522e2670eb1e9384b27b95fab05 not found: ID does not exist"
Nov 02 12:27:52.239947 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.239902    5879 scope.go:122] "RemoveContainer" containerID="efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"
Nov 02 12:27:52.240114 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.240099    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83"} err="failed to get container status \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": rpc error: code = NotFound desc = could not find container \"efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83\": container with ID starting with efc4b686b83c2be0f870af357429c446bd8d228f40ea5bcdde1bded207209a83 not found: ID does not exist"
Nov 02 12:27:52.240159 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.240115    5879 scope.go:122] "RemoveContainer" containerID="6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"
Nov 02 12:27:52.240312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.240297    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30"} err="failed to get container status \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": rpc error: code = NotFound desc = could not find container \"6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30\": container with ID starting with 6c9616cf1c9bdb18bc8430671587ee8def77ce56faaaf15e0194f7807f3ecc30 not found: ID does not exist"
Nov 02 12:27:52.240369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.240312    5879 scope.go:122] "RemoveContainer" containerID="64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"
Nov 02 12:27:52.240505 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.240487    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400"} err="failed to get container status \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": rpc error: code = NotFound desc = could not find container \"64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400\": container with ID starting with 64bd0c8111a75bd51c379c3c8fa45b1d5ac8ec720584bce04945f4ea8d3dd400 not found: ID does not exist"
Nov 02 12:27:52.240547 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.240506    5879 scope.go:122] "RemoveContainer" containerID="e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"
Nov 02 12:27:52.240717 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.240701    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64"} err="failed to get container status \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": rpc error: code = NotFound desc = could not find container \"e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64\": container with ID starting with e0c1acfcea29f3596bec222753c09bd6402060d317078de25e510f7baaa3fb64 not found: ID does not exist"
Nov 02 12:27:52.248508 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.248483    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 12:27:52.278245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278211    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 12:27:52.278647 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278631    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="kube-rbac-proxy"
Nov 02 12:27:52.278682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278648    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="kube-rbac-proxy"
Nov 02 12:27:52.278682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278664    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="kube-rbac-proxy-web"
Nov 02 12:27:52.278682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278669    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="kube-rbac-proxy-web"
Nov 02 12:27:52.278682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278676    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="prometheus"
Nov 02 12:27:52.278682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278682    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="prometheus"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278689    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="init-config-reloader"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278693    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="init-config-reloader"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278708    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="thanos-sidecar"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278712    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="thanos-sidecar"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278718    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="config-reloader"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278722    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="config-reloader"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278730    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="kube-rbac-proxy-thanos"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278735    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="kube-rbac-proxy-thanos"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278792    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="thanos-sidecar"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278799    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="config-reloader"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278805    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="kube-rbac-proxy-thanos"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278813    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="kube-rbac-proxy"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278819    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="kube-rbac-proxy-web"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278824    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="prometheus"
Nov 02 12:27:52.278836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.278829    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" containerName="init-config-reloader"
Nov 02 12:27:52.283154 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.283135    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299119 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299088    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299124    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299209    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299242    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299268    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299515 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299353    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-web-config\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299515 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299383    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299515 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299407    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299515 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299478    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-config\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299527    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299586    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b834743c-3e58-4455-bbad-5f1a0ee4ae50-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299615    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299644    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299705    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299757    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299799    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299844    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxbsj\" (UniqueName: \"kubernetes.io/projected/b834743c-3e58-4455-bbad-5f1a0ee4ae50-kube-api-access-dxbsj\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299884    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299916    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b834743c-3e58-4455-bbad-5f1a0ee4ae50-config-out\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.299956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.299946    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.300512 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.300495    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 12:27:52.300760 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.300742    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 12:27:52.300846 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.300779    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 12:27:52.300953 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.300921    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 12:27:52.301077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.300965    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 12:27:52.301077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.300966    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 12:27:52.301179 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.301075    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 12:27:52.301227 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.301186    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 12:27:52.301227 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.301207    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 12:27:52.301326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.301270    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 12:27:52.301767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.301754    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 12:27:52.301825 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.301786    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 12:27:52.301825 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.301815    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-v8788\""
Nov 02 12:27:52.301892 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.301825    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c8tgcn5qluvqu\""
Nov 02 12:27:52.316047 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.316014    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 12:27:52.316267 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.316247    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 12:27:52.316920 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.316898    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 12:27:52.401156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401121    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401160    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401441 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401178    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401441 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401200    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-web-config\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401441 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401344    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401441 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401381    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401441 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401400    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-config\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401441 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401422    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401457    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b834743c-3e58-4455-bbad-5f1a0ee4ae50-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401484    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401513    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401540    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401568    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401604    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401628    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxbsj\" (UniqueName: \"kubernetes.io/projected/b834743c-3e58-4455-bbad-5f1a0ee4ae50-kube-api-access-dxbsj\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401658    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401693    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b834743c-3e58-4455-bbad-5f1a0ee4ae50-config-out\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.401735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401722    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.402270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401753    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.402270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401786    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.402270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.401966    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.402419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.402357    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.402802 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.402777    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.402896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.402820    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.404666 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.404639    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.404767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.404639    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.404767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.404745    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.405224 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.405196    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.405334 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.405249    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-web-config\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.405405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.405383    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.406109 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.406080    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.406563 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.406542    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.407105 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.407087    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-config\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.407216 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.407142    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.407279 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.407260    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.407587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.407570    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b834743c-3e58-4455-bbad-5f1a0ee4ae50-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.407673 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.407656    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b834743c-3e58-4455-bbad-5f1a0ee4ae50-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.407846 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.407832    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b834743c-3e58-4455-bbad-5f1a0ee4ae50-config-out\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.408455 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.408439    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b834743c-3e58-4455-bbad-5f1a0ee4ae50-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.428817 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.428746    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxbsj\" (UniqueName: \"kubernetes.io/projected/b834743c-3e58-4455-bbad-5f1a0ee4ae50-kube-api-access-dxbsj\") pod \"prometheus-k8s-0\" (UID: \"b834743c-3e58-4455-bbad-5f1a0ee4ae50\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.601868 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.601826    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:27:52.954257 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:52.954229    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 12:27:52.957660 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:27:52.957623    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb834743c_3e58_4455_bbad_5f1a0ee4ae50.slice/crio-c1158efeebeea02c16ebd565724addee8a74c03c2bc7383fce9e9a6e87445657 WatchSource:0}: Error finding container c1158efeebeea02c16ebd565724addee8a74c03c2bc7383fce9e9a6e87445657: Status 404 returned error can't find the container with id c1158efeebeea02c16ebd565724addee8a74c03c2bc7383fce9e9a6e87445657
Nov 02 12:27:53.179857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:53.179820    5879 generic.go:356] "Generic (PLEG): container finished" podID="b834743c-3e58-4455-bbad-5f1a0ee4ae50" containerID="face84fb4809ac28eab13ac38763a9c99964c12cb824589e6e551fa25f34ff5b" exitCode=0
Nov 02 12:27:53.180376 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:53.179918    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b834743c-3e58-4455-bbad-5f1a0ee4ae50","Type":"ContainerDied","Data":"face84fb4809ac28eab13ac38763a9c99964c12cb824589e6e551fa25f34ff5b"}
Nov 02 12:27:53.180376 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:53.180023    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b834743c-3e58-4455-bbad-5f1a0ee4ae50","Type":"ContainerStarted","Data":"c1158efeebeea02c16ebd565724addee8a74c03c2bc7383fce9e9a6e87445657"}
Nov 02 12:27:54.013914 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:54.013882    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb2d74e-5ea9-48e3-98b8-53963e608786" path="/var/lib/kubelet/pods/ffb2d74e-5ea9-48e3-98b8-53963e608786/volumes"
Nov 02 12:27:54.147257 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:54.147223    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-26p8v"
Nov 02 12:27:54.187563 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:54.187533    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b834743c-3e58-4455-bbad-5f1a0ee4ae50","Type":"ContainerStarted","Data":"bbe1b7837793718b987cc139749488ddfed504bf4ecdf17279cd9ea4b650157e"}
Nov 02 12:27:54.188089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:54.187610    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b834743c-3e58-4455-bbad-5f1a0ee4ae50","Type":"ContainerStarted","Data":"ba5c070f33a6cae0f8a34a3bb8b29d395a8ca2e39055f0bebf859f87e9994721"}
Nov 02 12:27:54.188089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:54.187646    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b834743c-3e58-4455-bbad-5f1a0ee4ae50","Type":"ContainerStarted","Data":"31c6986ce2a6263b7df8f54142acf8dbaa1bb8a5434525a72281b0e0a65d7110"}
Nov 02 12:27:54.188089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:54.187688    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b834743c-3e58-4455-bbad-5f1a0ee4ae50","Type":"ContainerStarted","Data":"17e8526c41a62266e710b03ae17040efb4bd3667e5bec2b1e1e27e57a89a2d3a"}
Nov 02 12:27:54.188089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:54.187727    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b834743c-3e58-4455-bbad-5f1a0ee4ae50","Type":"ContainerStarted","Data":"f3a0c9831f30e056191a41aaf9b714cb438c1861e3b475a9eda96efae8f33744"}
Nov 02 12:27:54.188089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:54.187767    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b834743c-3e58-4455-bbad-5f1a0ee4ae50","Type":"ContainerStarted","Data":"3e8fa6f38f1e74d7e8ee37286abafe2190bf8da41263ed9a3bc9628fa533a76d"}
Nov 02 12:27:54.258903 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:54.258789    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.242775366 podStartE2EDuration="2.258775664s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.000298ms" isStatefulPod=false podCreationTimestamp="2026-11-02 12:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:27:54.257859395 +0000 UTC m=+20.364562246" watchObservedRunningTime="2026-11-02 12:27:54.258775664 +0000 UTC m=+20.365478514"
Nov 02 12:27:56.207104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:56.207067    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:27:57.602804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:27:57.602769    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:28:00.515721 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:00.515683    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:28:00.670694 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:00.670661    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:28:00.684219 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:00.684194    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:28:00.700652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:00.700623    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:28:01.231016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:01.230961    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 12:28:02.951950 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:02.951918    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2hzq7"
Nov 02 12:28:04.954283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:04.954244    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:28:05.082696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:05.082664    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:28:07.811939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:07.811903    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:28:07.936782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:07.936743    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:28:09.012437 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:09.012398    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:28:09.140294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:09.140260    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:28:14.821134 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:28:14.821091    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:28:24.781190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:24.781141    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:28:32.642371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:32.642339    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:28:44.820138 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:28:44.820096    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:28:57.728946 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:28:57.728907    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:14.819686 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:29:14.819606    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:29:28.456977 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:28.456926    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:28.602419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:28.602381    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:29.827053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:29.827013    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:29.957449 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:29.957412    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:31.045802 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:31.045772    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:32.769568 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:32.769539    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:33.782488 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:33.782458    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:33.902378 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:33.902347    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:35.024896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.024858    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:35.189313 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.189282    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:35.306869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.306781    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:35.409074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.409036    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"]
Nov 02 12:29:35.411407 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.411391    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:35.427335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.427306    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 02 12:29:35.441026 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.440984    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 02 12:29:35.441198 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.441101    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-zjdz9\""
Nov 02 12:29:35.441786 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.441768    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"]
Nov 02 12:29:35.532312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.532281    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw\" (UID: \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:35.532481 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.532326    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76xtr\" (UniqueName: \"kubernetes.io/projected/d9a539e3-e531-4b78-8759-e4cc4974ce2a-kube-api-access-76xtr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw\" (UID: \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:35.532481 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.532400    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw\" (UID: \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:35.633372 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.633338    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw\" (UID: \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:35.633538 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.633390    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76xtr\" (UniqueName: \"kubernetes.io/projected/d9a539e3-e531-4b78-8759-e4cc4974ce2a-kube-api-access-76xtr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw\" (UID: \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:35.633538 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.633419    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw\" (UID: \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:35.633819 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.633799    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw\" (UID: \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:35.633862 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.633801    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw\" (UID: \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:35.658582 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.658559    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76xtr\" (UniqueName: \"kubernetes.io/projected/d9a539e3-e531-4b78-8759-e4cc4974ce2a-kube-api-access-76xtr\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw\" (UID: \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:35.724674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:35.724642    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:36.019821 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:36.019793    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"]
Nov 02 12:29:36.022711 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:29:36.022684    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a539e3_e531_4b78_8759_e4cc4974ce2a.slice/crio-70d097196962f8904cca28574d1688be8a861c7f95a1563122d3a52b6b9c0fcd WatchSource:0}: Error finding container 70d097196962f8904cca28574d1688be8a861c7f95a1563122d3a52b6b9c0fcd: Status 404 returned error can't find the container with id 70d097196962f8904cca28574d1688be8a861c7f95a1563122d3a52b6b9c0fcd
Nov 02 12:29:36.024666 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:36.024645    5879 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:29:36.563125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:36.563083    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw" event={"ID":"d9a539e3-e531-4b78-8759-e4cc4974ce2a","Type":"ContainerStarted","Data":"70d097196962f8904cca28574d1688be8a861c7f95a1563122d3a52b6b9c0fcd"}
Nov 02 12:29:38.353131 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:38.353084    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:41.583954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:41.583857    5879 generic.go:356] "Generic (PLEG): container finished" podID="d9a539e3-e531-4b78-8759-e4cc4974ce2a" containerID="d4f267fe0cd77783485c199e1a9093f897ca3b3dbb8fc4b77cde5f9c88771556" exitCode=0
Nov 02 12:29:41.583954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:41.583896    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw" event={"ID":"d9a539e3-e531-4b78-8759-e4cc4974ce2a","Type":"ContainerDied","Data":"d4f267fe0cd77783485c199e1a9093f897ca3b3dbb8fc4b77cde5f9c88771556"}
Nov 02 12:29:44.597982 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:44.597944    5879 generic.go:356] "Generic (PLEG): container finished" podID="d9a539e3-e531-4b78-8759-e4cc4974ce2a" containerID="2c4a19b796d818e1a185143d288d04d80a69aaa6f8374cb09ab59ca6f82e3c14" exitCode=0
Nov 02 12:29:44.598384 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:44.598028    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw" event={"ID":"d9a539e3-e531-4b78-8759-e4cc4974ce2a","Type":"ContainerDied","Data":"2c4a19b796d818e1a185143d288d04d80a69aaa6f8374cb09ab59ca6f82e3c14"}
Nov 02 12:29:44.820263 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:29:44.820224    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:29:45.369926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:45.369887    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:51.630583 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:51.630548    5879 generic.go:356] "Generic (PLEG): container finished" podID="d9a539e3-e531-4b78-8759-e4cc4974ce2a" containerID="62d8653dd7a7c645f3c71122abfd062915950f7e9f2a830a6f4883859aee5eec" exitCode=0
Nov 02 12:29:51.630968 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:51.630627    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw" event={"ID":"d9a539e3-e531-4b78-8759-e4cc4974ce2a","Type":"ContainerDied","Data":"62d8653dd7a7c645f3c71122abfd062915950f7e9f2a830a6f4883859aee5eec"}
Nov 02 12:29:52.764714 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:52.764685    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:52.891831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:52.891738    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a539e3-e531-4b78-8759-e4cc4974ce2a-kube-api-access-76xtr\" (UniqueName: \"kubernetes.io/projected/d9a539e3-e531-4b78-8759-e4cc4974ce2a-kube-api-access-76xtr\") pod \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\" (UID: \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\") "
Nov 02 12:29:52.891831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:52.891823    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-util\" (UniqueName: \"kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-util\") pod \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\" (UID: \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\") "
Nov 02 12:29:52.892113 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:52.891916    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-bundle\") pod \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\" (UID: \"d9a539e3-e531-4b78-8759-e4cc4974ce2a\") "
Nov 02 12:29:52.892340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:52.892310    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-bundle" pod "d9a539e3-e531-4b78-8759-e4cc4974ce2a" (UID: "d9a539e3-e531-4b78-8759-e4cc4974ce2a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:52.894305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:52.894279    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a539e3-e531-4b78-8759-e4cc4974ce2a-kube-api-access-76xtr" pod "d9a539e3-e531-4b78-8759-e4cc4974ce2a" (UID: "d9a539e3-e531-4b78-8759-e4cc4974ce2a"). InnerVolumeSpecName "kube-api-access-76xtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:29:52.898022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:52.897960    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-util" pod "d9a539e3-e531-4b78-8759-e4cc4974ce2a" (UID: "d9a539e3-e531-4b78-8759-e4cc4974ce2a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:29:52.992647 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:52.992610    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76xtr\" (UniqueName: \"kubernetes.io/projected/d9a539e3-e531-4b78-8759-e4cc4974ce2a-kube-api-access-76xtr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:29:52.992647 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:52.992640    5879 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-util\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:29:52.992647 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:52.992650    5879 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9a539e3-e531-4b78-8759-e4cc4974ce2a-bundle\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:29:53.641130 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:53.641086    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw" event={"ID":"d9a539e3-e531-4b78-8759-e4cc4974ce2a","Type":"ContainerDied","Data":"70d097196962f8904cca28574d1688be8a861c7f95a1563122d3a52b6b9c0fcd"}
Nov 02 12:29:53.641130 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:53.641122    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6448z7pw"
Nov 02 12:29:53.641130 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:53.641135    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70d097196962f8904cca28574d1688be8a861c7f95a1563122d3a52b6b9c0fcd"
Nov 02 12:29:55.641228 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:55.641181    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:55.761964 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:55.761925    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:29:57.350037 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.350004    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p"]
Nov 02 12:29:57.350472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.350360    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d9a539e3-e531-4b78-8759-e4cc4974ce2a" containerName="extract"
Nov 02 12:29:57.350472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.350371    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a539e3-e531-4b78-8759-e4cc4974ce2a" containerName="extract"
Nov 02 12:29:57.350472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.350386    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d9a539e3-e531-4b78-8759-e4cc4974ce2a" containerName="util"
Nov 02 12:29:57.350472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.350391    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a539e3-e531-4b78-8759-e4cc4974ce2a" containerName="util"
Nov 02 12:29:57.350472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.350399    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d9a539e3-e531-4b78-8759-e4cc4974ce2a" containerName="pull"
Nov 02 12:29:57.350472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.350404    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a539e3-e531-4b78-8759-e4cc4974ce2a" containerName="pull"
Nov 02 12:29:57.350472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.350452    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a539e3-e531-4b78-8759-e4cc4974ce2a" containerName="extract"
Nov 02 12:29:57.352425 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.352408    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p"
Nov 02 12:29:57.382029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.381986    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 12:29:57.382174 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.382028    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 12:29:57.382174 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.382034    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-777lg\""
Nov 02 12:29:57.382757 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.382738    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p"]
Nov 02 12:29:57.534941 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.534905    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgb5q\" (UniqueName: \"kubernetes.io/projected/c838dea4-d2d3-4fe0-ad9f-79565b9473d6-kube-api-access-hgb5q\") pod \"jobset-operator-674cd59dc5-ghq5p\" (UID: \"c838dea4-d2d3-4fe0-ad9f-79565b9473d6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p"
Nov 02 12:29:57.535125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.534946    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c838dea4-d2d3-4fe0-ad9f-79565b9473d6-tmp\") pod \"jobset-operator-674cd59dc5-ghq5p\" (UID: \"c838dea4-d2d3-4fe0-ad9f-79565b9473d6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p"
Nov 02 12:29:57.635956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.635927    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgb5q\" (UniqueName: \"kubernetes.io/projected/c838dea4-d2d3-4fe0-ad9f-79565b9473d6-kube-api-access-hgb5q\") pod \"jobset-operator-674cd59dc5-ghq5p\" (UID: \"c838dea4-d2d3-4fe0-ad9f-79565b9473d6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p"
Nov 02 12:29:57.636142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.635967    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c838dea4-d2d3-4fe0-ad9f-79565b9473d6-tmp\") pod \"jobset-operator-674cd59dc5-ghq5p\" (UID: \"c838dea4-d2d3-4fe0-ad9f-79565b9473d6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p"
Nov 02 12:29:57.636389 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.636371    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c838dea4-d2d3-4fe0-ad9f-79565b9473d6-tmp\") pod \"jobset-operator-674cd59dc5-ghq5p\" (UID: \"c838dea4-d2d3-4fe0-ad9f-79565b9473d6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p"
Nov 02 12:29:57.661782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.661753    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgb5q\" (UniqueName: \"kubernetes.io/projected/c838dea4-d2d3-4fe0-ad9f-79565b9473d6-kube-api-access-hgb5q\") pod \"jobset-operator-674cd59dc5-ghq5p\" (UID: \"c838dea4-d2d3-4fe0-ad9f-79565b9473d6\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p"
Nov 02 12:29:57.672630 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.672601    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p"
Nov 02 12:29:57.973429 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:57.973404    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p"]
Nov 02 12:29:57.975363 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:29:57.975330    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc838dea4_d2d3_4fe0_ad9f_79565b9473d6.slice/crio-f1623cff896f6ead05dc7b2d470518c6a421e35a876df6feed6df315a76b4b92 WatchSource:0}: Error finding container f1623cff896f6ead05dc7b2d470518c6a421e35a876df6feed6df315a76b4b92: Status 404 returned error can't find the container with id f1623cff896f6ead05dc7b2d470518c6a421e35a876df6feed6df315a76b4b92
Nov 02 12:29:58.663396 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:29:58.663355    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p" event={"ID":"c838dea4-d2d3-4fe0-ad9f-79565b9473d6","Type":"ContainerStarted","Data":"f1623cff896f6ead05dc7b2d470518c6a421e35a876df6feed6df315a76b4b92"}
Nov 02 12:30:00.678038 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:00.677975    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p" event={"ID":"c838dea4-d2d3-4fe0-ad9f-79565b9473d6","Type":"ContainerStarted","Data":"f010c192b05146f12c1c65e5f0bbf20a7f84992041cd1388d87a4f89e482440e"}
Nov 02 12:30:00.721124 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:00.721067    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-ghq5p" podStartSLOduration=1.962454801 podStartE2EDuration="3.721052344s" totalImagesPullingTime="1.758597543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:00.720353102 +0000 UTC m=+146.827055966" watchObservedRunningTime="2026-11-02 12:30:00.721052344 +0000 UTC m=+146.827755194"
Nov 02 12:30:09.189234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:09.189191    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:10.846944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:10.846906    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:12.778949 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:12.778913    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:12.900977 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:12.900937    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:13.816346 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:13.816296    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:14.818771 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:30:14.818727    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:30:15.400199 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.400164    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:15.536897 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.536852    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:15.636892 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.636861    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"]
Nov 02 12:30:15.639231 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.639215    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"
Nov 02 12:30:15.661762 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.661699    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"]
Nov 02 12:30:15.674299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.674275    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4rx7t\""
Nov 02 12:30:15.674438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.674331    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 12:30:15.674438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.674361    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 12:30:15.684628 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.684604    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xftnt\" (UniqueName: \"kubernetes.io/projected/0a637f5d-3a2d-4672-9039-261f8d3e858a-kube-api-access-xftnt\") pod \"cert-manager-webhook-5dbbbf75cc-864lb\" (UID: \"0a637f5d-3a2d-4672-9039-261f8d3e858a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"
Nov 02 12:30:15.684742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.684635    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a637f5d-3a2d-4672-9039-261f8d3e858a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-864lb\" (UID: \"0a637f5d-3a2d-4672-9039-261f8d3e858a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"
Nov 02 12:30:15.785556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.785521    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xftnt\" (UniqueName: \"kubernetes.io/projected/0a637f5d-3a2d-4672-9039-261f8d3e858a-kube-api-access-xftnt\") pod \"cert-manager-webhook-5dbbbf75cc-864lb\" (UID: \"0a637f5d-3a2d-4672-9039-261f8d3e858a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"
Nov 02 12:30:15.785556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.785563    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a637f5d-3a2d-4672-9039-261f8d3e858a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-864lb\" (UID: \"0a637f5d-3a2d-4672-9039-261f8d3e858a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"
Nov 02 12:30:15.810513 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.810479    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0a637f5d-3a2d-4672-9039-261f8d3e858a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-864lb\" (UID: \"0a637f5d-3a2d-4672-9039-261f8d3e858a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"
Nov 02 12:30:15.810653 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.810496    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xftnt\" (UniqueName: \"kubernetes.io/projected/0a637f5d-3a2d-4672-9039-261f8d3e858a-kube-api-access-xftnt\") pod \"cert-manager-webhook-5dbbbf75cc-864lb\" (UID: \"0a637f5d-3a2d-4672-9039-261f8d3e858a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"
Nov 02 12:30:15.949233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:15.949140    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"
Nov 02 12:30:16.245098 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:16.245075    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"]
Nov 02 12:30:16.246904 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:30:16.246876    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a637f5d_3a2d_4672_9039_261f8d3e858a.slice/crio-6f81c435ab1d9d5cfd432f6a9aeac3124a8646e6176b5a7d7bd72ecd7b53ba59 WatchSource:0}: Error finding container 6f81c435ab1d9d5cfd432f6a9aeac3124a8646e6176b5a7d7bd72ecd7b53ba59: Status 404 returned error can't find the container with id 6f81c435ab1d9d5cfd432f6a9aeac3124a8646e6176b5a7d7bd72ecd7b53ba59
Nov 02 12:30:16.358520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:16.358478    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:16.741865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:16.741831    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb" event={"ID":"0a637f5d-3a2d-4672-9039-261f8d3e858a","Type":"ContainerStarted","Data":"6f81c435ab1d9d5cfd432f6a9aeac3124a8646e6176b5a7d7bd72ecd7b53ba59"}
Nov 02 12:30:17.596376 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:17.596335    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:17.821566 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:17.821524    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:17.976836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:17.976796    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:18.745176 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:18.745132    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:19.763340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:19.763254    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb" event={"ID":"0a637f5d-3a2d-4672-9039-261f8d3e858a","Type":"ContainerStarted","Data":"7819d4f800e213fe893c1f4016f93701614fa2c81fc2056ab17a650bc78e4759"}
Nov 02 12:30:19.763776 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:19.763396    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"
Nov 02 12:30:19.806464 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:19.806296    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb" podStartSLOduration=1.5861360370000002 podStartE2EDuration="4.806277772s" totalImagesPullingTime="3.220141735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:19.806193534 +0000 UTC m=+165.912896385" watchObservedRunningTime="2026-11-02 12:30:19.806277772 +0000 UTC m=+165.912980624"
Nov 02 12:30:21.851755 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:21.851702    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:21.977791 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:21.977757    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:22.781961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:22.781926    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:22.966799 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:22.966729    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:23.480351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:23.480319    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:23.780689 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:23.780585    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:23.917883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:23.917836    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:23.922807 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:23.922778    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"]
Nov 02 12:30:23.942344 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:23.942322    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:23.953592 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:23.953561    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"]
Nov 02 12:30:24.003869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.003837    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 12:30:24.003869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.003856    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wwbqf\""
Nov 02 12:30:24.003869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.003856    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 12:30:24.004403 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.003928    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 12:30:24.007099 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.007071    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 12:30:24.007099 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.007093    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 12:30:24.065062 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.064947    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/73134359-f199-481f-b3ec-8325f3d6e8a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.065062 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.065017    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fnnr\" (UniqueName: \"kubernetes.io/projected/73134359-f199-481f-b3ec-8325f3d6e8a2-kube-api-access-7fnnr\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.065337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.065135    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73134359-f199-481f-b3ec-8325f3d6e8a2-tmp\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.065337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.065182    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/73134359-f199-481f-b3ec-8325f3d6e8a2-metrics-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.065337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.065303    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73134359-f199-481f-b3ec-8325f3d6e8a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.065493 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.065349    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/73134359-f199-481f-b3ec-8325f3d6e8a2-manifests\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.098316 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.098282    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:24.166875 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.166838    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/73134359-f199-481f-b3ec-8325f3d6e8a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.167129 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.166888    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnnr\" (UniqueName: \"kubernetes.io/projected/73134359-f199-481f-b3ec-8325f3d6e8a2-kube-api-access-7fnnr\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.167129 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.167045    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73134359-f199-481f-b3ec-8325f3d6e8a2-tmp\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.167129 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.167117    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/73134359-f199-481f-b3ec-8325f3d6e8a2-metrics-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.167262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.167150    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73134359-f199-481f-b3ec-8325f3d6e8a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.167262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.167173    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/73134359-f199-481f-b3ec-8325f3d6e8a2-manifests\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.167570 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.167542    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/73134359-f199-481f-b3ec-8325f3d6e8a2-manifests\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.169571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.169541    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73134359-f199-481f-b3ec-8325f3d6e8a2-tmp\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.169673 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.169577    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73134359-f199-481f-b3ec-8325f3d6e8a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.169717 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.169700    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/73134359-f199-481f-b3ec-8325f3d6e8a2-metrics-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.169853 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.169832    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/73134359-f199-481f-b3ec-8325f3d6e8a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.194206 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.194179    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fnnr\" (UniqueName: \"kubernetes.io/projected/73134359-f199-481f-b3ec-8325f3d6e8a2-kube-api-access-7fnnr\") pod \"opendatahub-operator-controller-manager-67b54f6f58-jvdlq\" (UID: \"73134359-f199-481f-b3ec-8325f3d6e8a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.229884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.229843    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:24.279964 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.279925    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:24.363975 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.363930    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:24.488130 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.488093    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:24.645654 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.645629    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"]
Nov 02 12:30:24.646485 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:30:24.646462    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73134359_f199_481f_b3ec_8325f3d6e8a2.slice/crio-814f994f37637b38c0f7e45cbbb5f9c2012e85c46d405cd7f19878ac0c707a47 WatchSource:0}: Error finding container 814f994f37637b38c0f7e45cbbb5f9c2012e85c46d405cd7f19878ac0c707a47: Status 404 returned error can't find the container with id 814f994f37637b38c0f7e45cbbb5f9c2012e85c46d405cd7f19878ac0c707a47
Nov 02 12:30:24.783396 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:24.783361    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq" event={"ID":"73134359-f199-481f-b3ec-8325f3d6e8a2","Type":"ContainerStarted","Data":"814f994f37637b38c0f7e45cbbb5f9c2012e85c46d405cd7f19878ac0c707a47"}
Nov 02 12:30:25.606199 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:25.606169    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:25.770489 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:25.770461    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-864lb"
Nov 02 12:30:26.417574 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:26.417528    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:26.726912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:26.726823    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-htlgr"]
Nov 02 12:30:26.729544 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:26.729517    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-htlgr"
Nov 02 12:30:26.762829 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:26.762786    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ts4jh\""
Nov 02 12:30:26.763896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:26.763854    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-htlgr"]
Nov 02 12:30:26.800260 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:26.800218    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2trr4\" (UniqueName: \"kubernetes.io/projected/e90320c0-8af5-48c6-8054-7ee2e109b6d0-kube-api-access-2trr4\") pod \"cert-manager-597b95cb84-htlgr\" (UID: \"e90320c0-8af5-48c6-8054-7ee2e109b6d0\") " pod="cert-manager/cert-manager-597b95cb84-htlgr"
Nov 02 12:30:26.800457 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:26.800402    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e90320c0-8af5-48c6-8054-7ee2e109b6d0-bound-sa-token\") pod \"cert-manager-597b95cb84-htlgr\" (UID: \"e90320c0-8af5-48c6-8054-7ee2e109b6d0\") " pod="cert-manager/cert-manager-597b95cb84-htlgr"
Nov 02 12:30:26.901434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:26.901399    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2trr4\" (UniqueName: \"kubernetes.io/projected/e90320c0-8af5-48c6-8054-7ee2e109b6d0-kube-api-access-2trr4\") pod \"cert-manager-597b95cb84-htlgr\" (UID: \"e90320c0-8af5-48c6-8054-7ee2e109b6d0\") " pod="cert-manager/cert-manager-597b95cb84-htlgr"
Nov 02 12:30:26.901707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:26.901521    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e90320c0-8af5-48c6-8054-7ee2e109b6d0-bound-sa-token\") pod \"cert-manager-597b95cb84-htlgr\" (UID: \"e90320c0-8af5-48c6-8054-7ee2e109b6d0\") " pod="cert-manager/cert-manager-597b95cb84-htlgr"
Nov 02 12:30:26.928131 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:26.928100    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2trr4\" (UniqueName: \"kubernetes.io/projected/e90320c0-8af5-48c6-8054-7ee2e109b6d0-kube-api-access-2trr4\") pod \"cert-manager-597b95cb84-htlgr\" (UID: \"e90320c0-8af5-48c6-8054-7ee2e109b6d0\") " pod="cert-manager/cert-manager-597b95cb84-htlgr"
Nov 02 12:30:26.931584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:26.931556    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e90320c0-8af5-48c6-8054-7ee2e109b6d0-bound-sa-token\") pod \"cert-manager-597b95cb84-htlgr\" (UID: \"e90320c0-8af5-48c6-8054-7ee2e109b6d0\") " pod="cert-manager/cert-manager-597b95cb84-htlgr"
Nov 02 12:30:26.986653 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:26.986570    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:27.043158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:27.043122    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-htlgr"
Nov 02 12:30:27.353092 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:27.352898    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-htlgr"]
Nov 02 12:30:27.354807 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:30:27.354775    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode90320c0_8af5_48c6_8054_7ee2e109b6d0.slice/crio-c8cfb7645e5cf174e5738a206b28acef3359e04194ea9a80e62ddb4e894eaee5 WatchSource:0}: Error finding container c8cfb7645e5cf174e5738a206b28acef3359e04194ea9a80e62ddb4e894eaee5: Status 404 returned error can't find the container with id c8cfb7645e5cf174e5738a206b28acef3359e04194ea9a80e62ddb4e894eaee5
Nov 02 12:30:27.799874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:27.799837    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq" event={"ID":"73134359-f199-481f-b3ec-8325f3d6e8a2","Type":"ContainerStarted","Data":"368ec5aa5803d3fad93f809102ef5571928dc7a28ba0fd1275b5014063871d21"}
Nov 02 12:30:27.801344 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:27.801324    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-htlgr" event={"ID":"e90320c0-8af5-48c6-8054-7ee2e109b6d0","Type":"ContainerStarted","Data":"48ffe4b51c3cac6add50f9706086e9feebe664f7fe43d0c06d5e088daf63d84e"}
Nov 02 12:30:27.801455 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:27.801351    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-htlgr" event={"ID":"e90320c0-8af5-48c6-8054-7ee2e109b6d0","Type":"ContainerStarted","Data":"c8cfb7645e5cf174e5738a206b28acef3359e04194ea9a80e62ddb4e894eaee5"}
Nov 02 12:30:27.920678 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:27.920614    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-htlgr" podStartSLOduration=1.920600035 podStartE2EDuration="1.920600035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:27.92051958 +0000 UTC m=+174.027222430" watchObservedRunningTime="2026-11-02 12:30:27.920600035 +0000 UTC m=+174.027302885"
Nov 02 12:30:27.981839 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:27.981797    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:28.125931 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:28.125901    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:28.525174 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:28.525089    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:28.672351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:28.672319    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:28.807173 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:28.807087    5879 generic.go:356] "Generic (PLEG): container finished" podID="73134359-f199-481f-b3ec-8325f3d6e8a2" containerID="368ec5aa5803d3fad93f809102ef5571928dc7a28ba0fd1275b5014063871d21" exitCode=0
Nov 02 12:30:28.807576 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:28.807180    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq" event={"ID":"73134359-f199-481f-b3ec-8325f3d6e8a2","Type":"ContainerDied","Data":"368ec5aa5803d3fad93f809102ef5571928dc7a28ba0fd1275b5014063871d21"}
Nov 02 12:30:29.160071 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:29.160040    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:29.781422 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:29.781378    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:29.813632 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:29.813601    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq" event={"ID":"73134359-f199-481f-b3ec-8325f3d6e8a2","Type":"ContainerStarted","Data":"b85a2fd060c71c528759bb2398f5fad75373b4536cebea7d0ddb9fc310688793"}
Nov 02 12:30:29.814061 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:29.813742    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:29.880555 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:29.880497    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq" podStartSLOduration=3.045726081 podStartE2EDuration="6.880481884s" totalImagesPullingTime="3.259276039s" totalInitContainerRuntime="575.479764ms" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:29.878828942 +0000 UTC m=+175.985531796" watchObservedRunningTime="2026-11-02 12:30:29.880481884 +0000 UTC m=+175.987184808"
Nov 02 12:30:30.787841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:30.787807    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:32.877399 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:32.877367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:33.559819 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:33.559787    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:35.132521 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.132488    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"]
Nov 02 12:30:35.136914 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.136896    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"
Nov 02 12:30:35.157202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.157162    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"]
Nov 02 12:30:35.169728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.169700    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 02 12:30:35.169728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.169710    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 12:30:35.169728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.169723    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 12:30:35.169961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.169762    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nbd9h\""
Nov 02 12:30:35.169961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.169781    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 02 12:30:35.172900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.172874    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:35.284170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.284139    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rdc2r\" (UID: \"d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"
Nov 02 12:30:35.284357 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.284175    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tsj5\" (UniqueName: \"kubernetes.io/projected/d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb-kube-api-access-2tsj5\") pod \"kuadrant-console-plugin-66c4fb575d-rdc2r\" (UID: \"d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"
Nov 02 12:30:35.284357 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.284267    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rdc2r\" (UID: \"d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"
Nov 02 12:30:35.341968 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.341924    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:35.385166 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.385050    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rdc2r\" (UID: \"d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"
Nov 02 12:30:35.385166 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.385107    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tsj5\" (UniqueName: \"kubernetes.io/projected/d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb-kube-api-access-2tsj5\") pod \"kuadrant-console-plugin-66c4fb575d-rdc2r\" (UID: \"d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"
Nov 02 12:30:35.385431 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.385177    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rdc2r\" (UID: \"d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"
Nov 02 12:30:35.385934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.385893    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-rdc2r\" (UID: \"d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"
Nov 02 12:30:35.388623 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.388589    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-rdc2r\" (UID: \"d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"
Nov 02 12:30:35.440707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.440667    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tsj5\" (UniqueName: \"kubernetes.io/projected/d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb-kube-api-access-2tsj5\") pod \"kuadrant-console-plugin-66c4fb575d-rdc2r\" (UID: \"d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"
Nov 02 12:30:35.447606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.447559    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"
Nov 02 12:30:35.618430 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.618388    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:35.751427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.751405    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r"]
Nov 02 12:30:35.753145 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:30:35.753110    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6668a9b_ef69_4bfc_a4a4_5ff66c0b62bb.slice/crio-efd2c2473eb2ba9b02d8e8a39957f494d3d440e66738702c276cdbcdd85e3709 WatchSource:0}: Error finding container efd2c2473eb2ba9b02d8e8a39957f494d3d440e66738702c276cdbcdd85e3709: Status 404 returned error can't find the container with id efd2c2473eb2ba9b02d8e8a39957f494d3d440e66738702c276cdbcdd85e3709
Nov 02 12:30:35.839506 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:35.839471    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r" event={"ID":"d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb","Type":"ContainerStarted","Data":"efd2c2473eb2ba9b02d8e8a39957f494d3d440e66738702c276cdbcdd85e3709"}
Nov 02 12:30:36.727223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:36.727179    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:36.847550 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:36.847518    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:37.238749 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:37.238706    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:37.354181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:37.354139    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:38.964804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:38.964771    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:39.105817 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:39.105781    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:39.581502 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:39.581440    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:39.719308 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:39.719276    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:39.973128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:39.973100    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:40.095889 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:40.095853    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:40.820098 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:40.819988    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-67b54f6f58-jvdlq"
Nov 02 12:30:40.871396 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:40.871366    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r" event={"ID":"d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb","Type":"ContainerStarted","Data":"fc5c547b0f0f3256b21705c114d9499b4a18e4dda4d85156b27eda3c7251ee68"}
Nov 02 12:30:40.955680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:40.955628    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-rdc2r" podStartSLOduration=1.251556879 podStartE2EDuration="5.9556135s" totalImagesPullingTime="4.704056621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:40.954619122 +0000 UTC m=+187.061321973" watchObservedRunningTime="2026-11-02 12:30:40.9556135 +0000 UTC m=+187.062316351"
Nov 02 12:30:44.537573 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:44.537487    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:44.663347 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:44.663313    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:44.819335 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:30:44.819263    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:30:46.387257 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:46.387210    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:46.526338 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:46.526300    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:51.177907 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:51.177866    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:51.312215 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:51.312184    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:52.858529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:52.858492    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:52.986976 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:52.986940    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:53.192571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.192538    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:53.336805 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.336769    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:53.600178 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.600095    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"]
Nov 02 12:30:53.605132 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.605111    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"
Nov 02 12:30:53.618125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.618093    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:53.628533 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.628506    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"]
Nov 02 12:30:53.640830 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.640802    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 12:30:53.641029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.640834    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 12:30:53.649254 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.649233    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/def76310-095a-4ed5-a40d-983f53799e40-tmp\") pod \"kube-auth-proxy-9c9fcfcf5-hjsfn\" (UID: \"def76310-095a-4ed5-a40d-983f53799e40\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"
Nov 02 12:30:53.649422 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.649263    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22kvn\" (UniqueName: \"kubernetes.io/projected/def76310-095a-4ed5-a40d-983f53799e40-kube-api-access-22kvn\") pod \"kube-auth-proxy-9c9fcfcf5-hjsfn\" (UID: \"def76310-095a-4ed5-a40d-983f53799e40\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"
Nov 02 12:30:53.649422 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.649302    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def76310-095a-4ed5-a40d-983f53799e40-tls-certs\") pod \"kube-auth-proxy-9c9fcfcf5-hjsfn\" (UID: \"def76310-095a-4ed5-a40d-983f53799e40\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"
Nov 02 12:30:53.697747 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.697701    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 12:30:53.697747 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.697741    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 12:30:53.697937 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.697743    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2qsjb\""
Nov 02 12:30:53.750722 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.750686    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/def76310-095a-4ed5-a40d-983f53799e40-tmp\") pod \"kube-auth-proxy-9c9fcfcf5-hjsfn\" (UID: \"def76310-095a-4ed5-a40d-983f53799e40\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"
Nov 02 12:30:53.750722 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.750723    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22kvn\" (UniqueName: \"kubernetes.io/projected/def76310-095a-4ed5-a40d-983f53799e40-kube-api-access-22kvn\") pod \"kube-auth-proxy-9c9fcfcf5-hjsfn\" (UID: \"def76310-095a-4ed5-a40d-983f53799e40\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"
Nov 02 12:30:53.750922 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.750773    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def76310-095a-4ed5-a40d-983f53799e40-tls-certs\") pod \"kube-auth-proxy-9c9fcfcf5-hjsfn\" (UID: \"def76310-095a-4ed5-a40d-983f53799e40\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"
Nov 02 12:30:53.753568 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.753514    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/def76310-095a-4ed5-a40d-983f53799e40-tmp\") pod \"kube-auth-proxy-9c9fcfcf5-hjsfn\" (UID: \"def76310-095a-4ed5-a40d-983f53799e40\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"
Nov 02 12:30:53.753756 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.753728    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def76310-095a-4ed5-a40d-983f53799e40-tls-certs\") pod \"kube-auth-proxy-9c9fcfcf5-hjsfn\" (UID: \"def76310-095a-4ed5-a40d-983f53799e40\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"
Nov 02 12:30:53.770413 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.770382    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:53.777454 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.777430    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22kvn\" (UniqueName: \"kubernetes.io/projected/def76310-095a-4ed5-a40d-983f53799e40-kube-api-access-22kvn\") pod \"kube-auth-proxy-9c9fcfcf5-hjsfn\" (UID: \"def76310-095a-4ed5-a40d-983f53799e40\") " pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"
Nov 02 12:30:53.897826 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.897788    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:53.916673 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:53.916641    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"
Nov 02 12:30:54.020558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.020524    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:54.218290 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.218262    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn"]
Nov 02 12:30:54.221069 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:30:54.221042    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef76310_095a_4ed5_a40d_983f53799e40.slice/crio-ee1a012ab8231d10341265780c8a22761e8fd396bef72ff98f6a33aa55234815 WatchSource:0}: Error finding container ee1a012ab8231d10341265780c8a22761e8fd396bef72ff98f6a33aa55234815: Status 404 returned error can't find the container with id ee1a012ab8231d10341265780c8a22761e8fd396bef72ff98f6a33aa55234815
Nov 02 12:30:54.559238 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.559162    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9hzth"]
Nov 02 12:30:54.562278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.562262    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth"
Nov 02 12:30:54.581553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.581513    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 12:30:54.582667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.582643    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:54.583014 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.582968    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9hzth"]
Nov 02 12:30:54.659867 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.659834    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2cfef690-e749-4ab2-85bd-e5a7caba917e-config-file\") pod \"limitador-limitador-74dfcfb894-9hzth\" (UID: \"2cfef690-e749-4ab2-85bd-e5a7caba917e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth"
Nov 02 12:30:54.660069 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.659895    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfgl\" (UniqueName: \"kubernetes.io/projected/2cfef690-e749-4ab2-85bd-e5a7caba917e-kube-api-access-bxfgl\") pod \"limitador-limitador-74dfcfb894-9hzth\" (UID: \"2cfef690-e749-4ab2-85bd-e5a7caba917e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth"
Nov 02 12:30:54.698888 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.698855    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9hzth"]
Nov 02 12:30:54.705508 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.705478    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:54.760444 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.760417    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2cfef690-e749-4ab2-85bd-e5a7caba917e-config-file\") pod \"limitador-limitador-74dfcfb894-9hzth\" (UID: \"2cfef690-e749-4ab2-85bd-e5a7caba917e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth"
Nov 02 12:30:54.760615 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.760464    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfgl\" (UniqueName: \"kubernetes.io/projected/2cfef690-e749-4ab2-85bd-e5a7caba917e-kube-api-access-bxfgl\") pod \"limitador-limitador-74dfcfb894-9hzth\" (UID: \"2cfef690-e749-4ab2-85bd-e5a7caba917e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth"
Nov 02 12:30:54.761075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.761057    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2cfef690-e749-4ab2-85bd-e5a7caba917e-config-file\") pod \"limitador-limitador-74dfcfb894-9hzth\" (UID: \"2cfef690-e749-4ab2-85bd-e5a7caba917e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth"
Nov 02 12:30:54.785929 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.785900    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfgl\" (UniqueName: \"kubernetes.io/projected/2cfef690-e749-4ab2-85bd-e5a7caba917e-kube-api-access-bxfgl\") pod \"limitador-limitador-74dfcfb894-9hzth\" (UID: \"2cfef690-e749-4ab2-85bd-e5a7caba917e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth"
Nov 02 12:30:54.831202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.831103    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:54.873957 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.873921    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth"
Nov 02 12:30:54.935106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:54.935073    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn" event={"ID":"def76310-095a-4ed5-a40d-983f53799e40","Type":"ContainerStarted","Data":"ee1a012ab8231d10341265780c8a22761e8fd396bef72ff98f6a33aa55234815"}
Nov 02 12:30:55.190213 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.189767    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9hzth"]
Nov 02 12:30:55.202049 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:30:55.200856    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cfef690_e749_4ab2_85bd_e5a7caba917e.slice/crio-deab30c67bd8fe12d539156776f15a784fc565df4524885168c885d121ba6b4c WatchSource:0}: Error finding container deab30c67bd8fe12d539156776f15a784fc565df4524885168c885d121ba6b4c: Status 404 returned error can't find the container with id deab30c67bd8fe12d539156776f15a784fc565df4524885168c885d121ba6b4c
Nov 02 12:30:55.313921 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.313882    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:55.517102 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.517004    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-x9j58"]
Nov 02 12:30:55.520474 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.520450    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-x9j58"
Nov 02 12:30:55.537186 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.537147    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x7vbt\""
Nov 02 12:30:55.538086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.538059    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-x9j58"]
Nov 02 12:30:55.569179 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.569139    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2rww\" (UniqueName: \"kubernetes.io/projected/e4591fa1-af64-4511-b696-a7d72297c156-kube-api-access-z2rww\") pod \"authorino-57f7f8659f-x9j58\" (UID: \"e4591fa1-af64-4511-b696-a7d72297c156\") " pod="kuadrant-system/authorino-57f7f8659f-x9j58"
Nov 02 12:30:55.612638 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.612591    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:55.670082 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.670049    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2rww\" (UniqueName: \"kubernetes.io/projected/e4591fa1-af64-4511-b696-a7d72297c156-kube-api-access-z2rww\") pod \"authorino-57f7f8659f-x9j58\" (UID: \"e4591fa1-af64-4511-b696-a7d72297c156\") " pod="kuadrant-system/authorino-57f7f8659f-x9j58"
Nov 02 12:30:55.695235 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.695200    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2rww\" (UniqueName: \"kubernetes.io/projected/e4591fa1-af64-4511-b696-a7d72297c156-kube-api-access-z2rww\") pod \"authorino-57f7f8659f-x9j58\" (UID: \"e4591fa1-af64-4511-b696-a7d72297c156\") " pod="kuadrant-system/authorino-57f7f8659f-x9j58"
Nov 02 12:30:55.736573 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.736534    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:55.835301 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.835217    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-x9j58"
Nov 02 12:30:55.942680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:55.942410    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth" event={"ID":"2cfef690-e749-4ab2-85bd-e5a7caba917e","Type":"ContainerStarted","Data":"deab30c67bd8fe12d539156776f15a784fc565df4524885168c885d121ba6b4c"}
Nov 02 12:30:56.141951 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:56.141916    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-x9j58"]
Nov 02 12:30:56.144490 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:30:56.144443    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4591fa1_af64_4511_b696_a7d72297c156.slice/crio-e327198f7e2268ccfce714845fb31dc82fd812fdc3dbf283ec4e750cfcdd83a4 WatchSource:0}: Error finding container e327198f7e2268ccfce714845fb31dc82fd812fdc3dbf283ec4e750cfcdd83a4: Status 404 returned error can't find the container with id e327198f7e2268ccfce714845fb31dc82fd812fdc3dbf283ec4e750cfcdd83a4
Nov 02 12:30:56.506469 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:56.506384    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:30:56.949383 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:56.949353    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-x9j58" event={"ID":"e4591fa1-af64-4511-b696-a7d72297c156","Type":"ContainerStarted","Data":"e327198f7e2268ccfce714845fb31dc82fd812fdc3dbf283ec4e750cfcdd83a4"}
Nov 02 12:30:56.951443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:56.951413    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn" event={"ID":"def76310-095a-4ed5-a40d-983f53799e40","Type":"ContainerStarted","Data":"76dff639aa362ed8921d1264052aa85b660caeaf33f008d59adc8aadd9649ce3"}
Nov 02 12:30:56.996505 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:56.996438    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9c9fcfcf5-hjsfn" podStartSLOduration=1.385117203 podStartE2EDuration="3.99641573s" totalImagesPullingTime="2.611298527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:56.995263565 +0000 UTC m=+203.101966417" watchObservedRunningTime="2026-11-02 12:30:56.99641573 +0000 UTC m=+203.103118578"
Nov 02 12:30:58.962526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:58.962485    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth" event={"ID":"2cfef690-e749-4ab2-85bd-e5a7caba917e","Type":"ContainerStarted","Data":"6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f"}
Nov 02 12:30:58.963051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:58.962758    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth"
Nov 02 12:30:59.013701 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:59.013652    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth" podStartSLOduration=2.176664012 podStartE2EDuration="5.013637605s" totalImagesPullingTime="2.836973593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:30:59.012634551 +0000 UTC m=+205.119337403" watchObservedRunningTime="2026-11-02 12:30:59.013637605 +0000 UTC m=+205.120340455"
Nov 02 12:30:59.969317 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:59.969265    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-x9j58" event={"ID":"e4591fa1-af64-4511-b696-a7d72297c156","Type":"ContainerStarted","Data":"e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3"}
Nov 02 12:30:59.995339 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:30:59.995299    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-x9j58"]
Nov 02 12:31:00.013202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:00.013150    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-x9j58" podStartSLOduration=1.664459454 podStartE2EDuration="5.013135547s" totalImagesPullingTime="3.348676093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:31:00.011508841 +0000 UTC m=+206.118211693" watchObservedRunningTime="2026-11-02 12:31:00.013135547 +0000 UTC m=+206.119838376"
Nov 02 12:31:00.240049 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:00.239950    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:00.366380 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:00.366350    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:00.610390 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:00.610303    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:00.729565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:00.729529    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:01.161104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:01.161069    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:01.294800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:01.294767    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:01.981917 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:01.981875    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-x9j58" podUID="e4591fa1-af64-4511-b696-a7d72297c156" containerName="authorino" containerID="cri-o://e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3" gracePeriod=30
Nov 02 12:31:02.224809 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.224784    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-x9j58"
Nov 02 12:31:02.334144 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.334042    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4591fa1-af64-4511-b696-a7d72297c156-kube-api-access-z2rww\" (UniqueName: \"kubernetes.io/projected/e4591fa1-af64-4511-b696-a7d72297c156-kube-api-access-z2rww\") pod \"e4591fa1-af64-4511-b696-a7d72297c156\" (UID: \"e4591fa1-af64-4511-b696-a7d72297c156\") "
Nov 02 12:31:02.336681 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.336645    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4591fa1-af64-4511-b696-a7d72297c156-kube-api-access-z2rww" pod "e4591fa1-af64-4511-b696-a7d72297c156" (UID: "e4591fa1-af64-4511-b696-a7d72297c156"). InnerVolumeSpecName "kube-api-access-z2rww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:31:02.434944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.434910    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2rww\" (UniqueName: \"kubernetes.io/projected/e4591fa1-af64-4511-b696-a7d72297c156-kube-api-access-z2rww\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:31:02.696408 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.696369    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"]
Nov 02 12:31:02.696826 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.696811    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e4591fa1-af64-4511-b696-a7d72297c156" containerName="authorino"
Nov 02 12:31:02.696866 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.696829    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4591fa1-af64-4511-b696-a7d72297c156" containerName="authorino"
Nov 02 12:31:02.696912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.696902    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4591fa1-af64-4511-b696-a7d72297c156" containerName="authorino"
Nov 02 12:31:02.699373 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.699335    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.729198 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.729169    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 12:31:02.729198 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.729199    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 12:31:02.729370 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.729208    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-r9ssx\""
Nov 02 12:31:02.757593 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.757565    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 02 12:31:02.758535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.758515    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"]
Nov 02 12:31:02.838315 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.838280    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b2da242-fe89-4999-a164-2f251fb97b3f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-klw92\" (UID: \"0b2da242-fe89-4999-a164-2f251fb97b3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.838315 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.838312    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b2da242-fe89-4999-a164-2f251fb97b3f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-klw92\" (UID: \"0b2da242-fe89-4999-a164-2f251fb97b3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.838504 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.838338    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5f5g\" (UniqueName: \"kubernetes.io/projected/0b2da242-fe89-4999-a164-2f251fb97b3f-kube-api-access-b5f5g\") pod \"kserve-module-controller-manager-7566cf5fc8-klw92\" (UID: \"0b2da242-fe89-4999-a164-2f251fb97b3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.838504 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.838463    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b2da242-fe89-4999-a164-2f251fb97b3f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-klw92\" (UID: \"0b2da242-fe89-4999-a164-2f251fb97b3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.939620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.939588    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b2da242-fe89-4999-a164-2f251fb97b3f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-klw92\" (UID: \"0b2da242-fe89-4999-a164-2f251fb97b3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.939620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.939619    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b2da242-fe89-4999-a164-2f251fb97b3f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-klw92\" (UID: \"0b2da242-fe89-4999-a164-2f251fb97b3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.939818 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.939643    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5f5g\" (UniqueName: \"kubernetes.io/projected/0b2da242-fe89-4999-a164-2f251fb97b3f-kube-api-access-b5f5g\") pod \"kserve-module-controller-manager-7566cf5fc8-klw92\" (UID: \"0b2da242-fe89-4999-a164-2f251fb97b3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.939818 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.939696    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b2da242-fe89-4999-a164-2f251fb97b3f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-klw92\" (UID: \"0b2da242-fe89-4999-a164-2f251fb97b3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.940083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.940063    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0b2da242-fe89-4999-a164-2f251fb97b3f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-klw92\" (UID: \"0b2da242-fe89-4999-a164-2f251fb97b3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.940125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.940079    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b2da242-fe89-4999-a164-2f251fb97b3f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-klw92\" (UID: \"0b2da242-fe89-4999-a164-2f251fb97b3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.940557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.940540    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0b2da242-fe89-4999-a164-2f251fb97b3f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-klw92\" (UID: \"0b2da242-fe89-4999-a164-2f251fb97b3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.964845 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.964786    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5f5g\" (UniqueName: \"kubernetes.io/projected/0b2da242-fe89-4999-a164-2f251fb97b3f-kube-api-access-b5f5g\") pod \"kserve-module-controller-manager-7566cf5fc8-klw92\" (UID: \"0b2da242-fe89-4999-a164-2f251fb97b3f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:02.987715 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.987683    5879 generic.go:356] "Generic (PLEG): container finished" podID="e4591fa1-af64-4511-b696-a7d72297c156" containerID="e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3" exitCode=0
Nov 02 12:31:02.987840 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.987729    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-x9j58"
Nov 02 12:31:02.987840 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.987748    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-x9j58" event={"ID":"e4591fa1-af64-4511-b696-a7d72297c156","Type":"ContainerDied","Data":"e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3"}
Nov 02 12:31:02.987840 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.987787    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-x9j58" event={"ID":"e4591fa1-af64-4511-b696-a7d72297c156","Type":"ContainerDied","Data":"e327198f7e2268ccfce714845fb31dc82fd812fdc3dbf283ec4e750cfcdd83a4"}
Nov 02 12:31:02.987840 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.987804    5879 scope.go:122] "RemoveContainer" containerID="e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3"
Nov 02 12:31:02.996434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.996416    5879 scope.go:122] "RemoveContainer" containerID="e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3"
Nov 02 12:31:02.996726 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:31:02.996705    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3\": container with ID starting with e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3 not found: ID does not exist" containerID="e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3"
Nov 02 12:31:02.996770 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:02.996736    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3"} err="failed to get container status \"e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3\": rpc error: code = NotFound desc = could not find container \"e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3\": container with ID starting with e68cbbebb90be308d594f4a9561e87c43a20ec73a99164a717d710d513fb00e3 not found: ID does not exist"
Nov 02 12:31:03.010766 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:03.010738    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:03.043393 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:03.043361    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-x9j58"]
Nov 02 12:31:03.046920 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:03.046891    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-x9j58"]
Nov 02 12:31:03.325732 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:03.325700    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"]
Nov 02 12:31:03.326386 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:31:03.326363    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2da242_fe89_4999_a164_2f251fb97b3f.slice/crio-6128c932cd9825b7d408b227b82c06d8240e10aaf4c03da666534d759cafb9e6 WatchSource:0}: Error finding container 6128c932cd9825b7d408b227b82c06d8240e10aaf4c03da666534d759cafb9e6: Status 404 returned error can't find the container with id 6128c932cd9825b7d408b227b82c06d8240e10aaf4c03da666534d759cafb9e6
Nov 02 12:31:03.995074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:03.995036    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92" event={"ID":"0b2da242-fe89-4999-a164-2f251fb97b3f","Type":"ContainerStarted","Data":"6128c932cd9825b7d408b227b82c06d8240e10aaf4c03da666534d759cafb9e6"}
Nov 02 12:31:04.016693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:04.016656    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4591fa1-af64-4511-b696-a7d72297c156" path="/var/lib/kubelet/pods/e4591fa1-af64-4511-b696-a7d72297c156/volumes"
Nov 02 12:31:04.915331 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:04.915292    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:05.919177 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:05.919135    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:06.005746 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:06.005710    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92" event={"ID":"0b2da242-fe89-4999-a164-2f251fb97b3f","Type":"ContainerStarted","Data":"62972c8e5254d28288d6e1e2e97700a01864e750a02ae7761499705b83bb2d06"}
Nov 02 12:31:06.005947 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:06.005794    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:06.067081 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:06.067035    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92" podStartSLOduration=2.40223548 podStartE2EDuration="4.067018862s" totalImagesPullingTime="1.664783382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:31:06.066464831 +0000 UTC m=+212.173167682" watchObservedRunningTime="2026-11-02 12:31:06.067018862 +0000 UTC m=+212.173721707"
Nov 02 12:31:07.273464 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:07.273427    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:07.399552 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:07.399518    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:09.970756 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:09.970720    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth"
Nov 02 12:31:10.961479 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:10.961447    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9hzth"]
Nov 02 12:31:10.961669 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:10.961647    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth" podUID="2cfef690-e749-4ab2-85bd-e5a7caba917e" containerName="limitador" containerID="cri-o://6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f" gracePeriod=30
Nov 02 12:31:11.513446 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:11.513421    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth"
Nov 02 12:31:11.620201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:11.620172    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cfef690-e749-4ab2-85bd-e5a7caba917e-config-file\" (UniqueName: \"kubernetes.io/configmap/2cfef690-e749-4ab2-85bd-e5a7caba917e-config-file\") pod \"2cfef690-e749-4ab2-85bd-e5a7caba917e\" (UID: \"2cfef690-e749-4ab2-85bd-e5a7caba917e\") "
Nov 02 12:31:11.620364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:11.620229    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cfef690-e749-4ab2-85bd-e5a7caba917e-kube-api-access-bxfgl\" (UniqueName: \"kubernetes.io/projected/2cfef690-e749-4ab2-85bd-e5a7caba917e-kube-api-access-bxfgl\") pod \"2cfef690-e749-4ab2-85bd-e5a7caba917e\" (UID: \"2cfef690-e749-4ab2-85bd-e5a7caba917e\") "
Nov 02 12:31:11.620572 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:11.620547    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cfef690-e749-4ab2-85bd-e5a7caba917e-config-file" pod "2cfef690-e749-4ab2-85bd-e5a7caba917e" (UID: "2cfef690-e749-4ab2-85bd-e5a7caba917e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:31:11.622436 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:11.622416    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cfef690-e749-4ab2-85bd-e5a7caba917e-kube-api-access-bxfgl" pod "2cfef690-e749-4ab2-85bd-e5a7caba917e" (UID: "2cfef690-e749-4ab2-85bd-e5a7caba917e"). InnerVolumeSpecName "kube-api-access-bxfgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:31:11.721121 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:11.721086    5879 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2cfef690-e749-4ab2-85bd-e5a7caba917e-config-file\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:31:11.721121 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:11.721115    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxfgl\" (UniqueName: \"kubernetes.io/projected/2cfef690-e749-4ab2-85bd-e5a7caba917e-kube-api-access-bxfgl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:31:12.032331 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:12.032300    5879 generic.go:356] "Generic (PLEG): container finished" podID="2cfef690-e749-4ab2-85bd-e5a7caba917e" containerID="6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f" exitCode=0
Nov 02 12:31:12.032460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:12.032340    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth"
Nov 02 12:31:12.032460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:12.032342    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth" event={"ID":"2cfef690-e749-4ab2-85bd-e5a7caba917e","Type":"ContainerDied","Data":"6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f"}
Nov 02 12:31:12.032460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:12.032451    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-9hzth" event={"ID":"2cfef690-e749-4ab2-85bd-e5a7caba917e","Type":"ContainerDied","Data":"deab30c67bd8fe12d539156776f15a784fc565df4524885168c885d121ba6b4c"}
Nov 02 12:31:12.032576 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:12.032474    5879 scope.go:122] "RemoveContainer" containerID="6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f"
Nov 02 12:31:12.041766 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:12.041748    5879 scope.go:122] "RemoveContainer" containerID="6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f"
Nov 02 12:31:12.042064 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:31:12.042042    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f\": container with ID starting with 6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f not found: ID does not exist" containerID="6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f"
Nov 02 12:31:12.042169 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:12.042072    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f"} err="failed to get container status \"6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f\": rpc error: code = NotFound desc = could not find container \"6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f\": container with ID starting with 6d0919a95b4be4ed6cea9f50319c91217af9206ed5a8d5a6794dae7e9359512f not found: ID does not exist"
Nov 02 12:31:12.066146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:12.066122    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:12.086683 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:12.086657    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9hzth"]
Nov 02 12:31:12.092293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:12.092265    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-9hzth"]
Nov 02 12:31:12.188282 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:12.188235    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:14.015006 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:14.014951    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cfef690-e749-4ab2-85bd-e5a7caba917e" path="/var/lib/kubelet/pods/2cfef690-e749-4ab2-85bd-e5a7caba917e/volumes"
Nov 02 12:31:14.524365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:14.524312    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:14.646094 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:14.646059    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:14.819317 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:31:14.819233    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:31:20.518137 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:20.518102    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:20.647145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:20.647112    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:21.107445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.107412    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"]
Nov 02 12:31:21.107807 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.107796    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2cfef690-e749-4ab2-85bd-e5a7caba917e" containerName="limitador"
Nov 02 12:31:21.107851 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.107809    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cfef690-e749-4ab2-85bd-e5a7caba917e" containerName="limitador"
Nov 02 12:31:21.107886 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.107868    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cfef690-e749-4ab2-85bd-e5a7caba917e" containerName="limitador"
Nov 02 12:31:21.110948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.110926    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"
Nov 02 12:31:21.129348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.129323    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 02 12:31:21.130351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.130330    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"]
Nov 02 12:31:21.142985 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.142956    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 12:31:21.142985 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.142962    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 12:31:21.216952 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.216915    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:21.219144 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.219115    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp552\" (UniqueName: \"kubernetes.io/projected/aae37103-208d-4ba4-a142-f5a29c9fdc37-kube-api-access-hp552\") pod \"postgres-7cdbf6dc44-kzdf5\" (UID: \"aae37103-208d-4ba4-a142-f5a29c9fdc37\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"
Nov 02 12:31:21.219302 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.219284    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aae37103-208d-4ba4-a142-f5a29c9fdc37-data\") pod \"postgres-7cdbf6dc44-kzdf5\" (UID: \"aae37103-208d-4ba4-a142-f5a29c9fdc37\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"
Nov 02 12:31:21.229883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.229860    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-8qxz4\""
Nov 02 12:31:21.319959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.319912    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aae37103-208d-4ba4-a142-f5a29c9fdc37-data\") pod \"postgres-7cdbf6dc44-kzdf5\" (UID: \"aae37103-208d-4ba4-a142-f5a29c9fdc37\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"
Nov 02 12:31:21.320135 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.320037    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp552\" (UniqueName: \"kubernetes.io/projected/aae37103-208d-4ba4-a142-f5a29c9fdc37-kube-api-access-hp552\") pod \"postgres-7cdbf6dc44-kzdf5\" (UID: \"aae37103-208d-4ba4-a142-f5a29c9fdc37\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"
Nov 02 12:31:21.320354 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.320335    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/aae37103-208d-4ba4-a142-f5a29c9fdc37-data\") pod \"postgres-7cdbf6dc44-kzdf5\" (UID: \"aae37103-208d-4ba4-a142-f5a29c9fdc37\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"
Nov 02 12:31:21.345097 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.345062    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp552\" (UniqueName: \"kubernetes.io/projected/aae37103-208d-4ba4-a142-f5a29c9fdc37-kube-api-access-hp552\") pod \"postgres-7cdbf6dc44-kzdf5\" (UID: \"aae37103-208d-4ba4-a142-f5a29c9fdc37\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"
Nov 02 12:31:21.423432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.423397    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"
Nov 02 12:31:21.470526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.470490    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:21.594520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.594486    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:21.719678 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.719384    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"]
Nov 02 12:31:21.721032 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:31:21.720983    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae37103_208d_4ba4_a142_f5a29c9fdc37.slice/crio-7204f8117489253c14c5e74448a6d1ecabb9785d8ab936822cff6da28a236f20 WatchSource:0}: Error finding container 7204f8117489253c14c5e74448a6d1ecabb9785d8ab936822cff6da28a236f20: Status 404 returned error can't find the container with id 7204f8117489253c14c5e74448a6d1ecabb9785d8ab936822cff6da28a236f20
Nov 02 12:31:21.875256 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:21.875223    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:22.016126 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:22.016047    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:22.076070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:22.076035    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5" event={"ID":"aae37103-208d-4ba4-a142-f5a29c9fdc37","Type":"ContainerStarted","Data":"7204f8117489253c14c5e74448a6d1ecabb9785d8ab936822cff6da28a236f20"}
Nov 02 12:31:22.276954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:22.276874    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:22.405307 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:22.405281    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:27.012338 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:27.012309    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-klw92"
Nov 02 12:31:28.680399 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:28.680362    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:28.811880 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:28.811829    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:32.995374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:32.995348    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:33.128775 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:33.128735    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5" event={"ID":"aae37103-208d-4ba4-a142-f5a29c9fdc37","Type":"ContainerStarted","Data":"89936f48bc7587bad201ba0e8569ebbdd3092cc26cc724962e4baeb5ce362cb4"}
Nov 02 12:31:33.128976 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:33.128916    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"
Nov 02 12:31:33.172185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:33.172132    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5" podStartSLOduration=0.937704102 podStartE2EDuration="12.172118341s" totalImagesPullingTime="11.234414239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:31:33.171567619 +0000 UTC m=+239.278270462" watchObservedRunningTime="2026-11-02 12:31:33.172118341 +0000 UTC m=+239.278821193"
Nov 02 12:31:33.489440 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:33.489402    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:33.643524 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:33.643491    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:35.957539 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:35.957503    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:36.089612 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:36.089567    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:39.164294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:39.164264    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzdf5"
Nov 02 12:31:44.820620 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:31:44.820572    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:31:46.250563 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.250533    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:46.306836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.306799    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-689ccbb994-jcbnx"]
Nov 02 12:31:46.310067 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.310042    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:31:46.326150 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.326130    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 02 12:31:46.339619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.339589    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 02 12:31:46.339767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.339644    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 02 12:31:46.339815 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.339793    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-crg4p\""
Nov 02 12:31:46.340945 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.340929    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-689ccbb994-jcbnx"]
Nov 02 12:31:46.374747 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.374711    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:46.459715 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.459674    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x742\" (UniqueName: \"kubernetes.io/projected/49a53aee-e53f-41f8-97eb-182cc35f94c2-kube-api-access-7x742\") pod \"maas-controller-689ccbb994-jcbnx\" (UID: \"49a53aee-e53f-41f8-97eb-182cc35f94c2\") " pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:31:46.459899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.459790    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-webhook-cert\") pod \"maas-controller-689ccbb994-jcbnx\" (UID: \"49a53aee-e53f-41f8-97eb-182cc35f94c2\") " pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:31:46.459899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.459875    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-metrics-certs\") pod \"maas-controller-689ccbb994-jcbnx\" (UID: \"49a53aee-e53f-41f8-97eb-182cc35f94c2\") " pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:31:46.499033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.499003    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:46.560967 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.560890    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-webhook-cert\") pod \"maas-controller-689ccbb994-jcbnx\" (UID: \"49a53aee-e53f-41f8-97eb-182cc35f94c2\") " pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:31:46.560967 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.560958    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-metrics-certs\") pod \"maas-controller-689ccbb994-jcbnx\" (UID: \"49a53aee-e53f-41f8-97eb-182cc35f94c2\") " pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:31:46.561171 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.560984    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x742\" (UniqueName: \"kubernetes.io/projected/49a53aee-e53f-41f8-97eb-182cc35f94c2-kube-api-access-7x742\") pod \"maas-controller-689ccbb994-jcbnx\" (UID: \"49a53aee-e53f-41f8-97eb-182cc35f94c2\") " pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:31:46.563556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.563528    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-metrics-certs\") pod \"maas-controller-689ccbb994-jcbnx\" (UID: \"49a53aee-e53f-41f8-97eb-182cc35f94c2\") " pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:31:46.563664 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.563559    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-webhook-cert\") pod \"maas-controller-689ccbb994-jcbnx\" (UID: \"49a53aee-e53f-41f8-97eb-182cc35f94c2\") " pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:31:46.591024 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.590979    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x742\" (UniqueName: \"kubernetes.io/projected/49a53aee-e53f-41f8-97eb-182cc35f94c2-kube-api-access-7x742\") pod \"maas-controller-689ccbb994-jcbnx\" (UID: \"49a53aee-e53f-41f8-97eb-182cc35f94c2\") " pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:31:46.613879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.613844    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:46.622682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.622662    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:31:46.832331 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.832249    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-blz6m"]
Nov 02 12:31:46.835079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.835062    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:31:46.856775 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.856742    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-blz6m"]
Nov 02 12:31:46.922243 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.922219    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-689ccbb994-jcbnx"]
Nov 02 12:31:46.924095 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:31:46.924067    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a53aee_e53f_41f8_97eb_182cc35f94c2.slice/crio-34a3937422bf0e58805401cfc9ff81b4e1f546ab4f2dcf0c61c20270fbcaf520 WatchSource:0}: Error finding container 34a3937422bf0e58805401cfc9ff81b4e1f546ab4f2dcf0c61c20270fbcaf520: Status 404 returned error can't find the container with id 34a3937422bf0e58805401cfc9ff81b4e1f546ab4f2dcf0c61c20270fbcaf520
Nov 02 12:31:46.964919 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.964887    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-webhook-cert\") pod \"maas-controller-5597f6b9f6-blz6m\" (UID: \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\") " pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:31:46.965107 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.964924    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbdp\" (UniqueName: \"kubernetes.io/projected/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-kube-api-access-wnbdp\") pod \"maas-controller-5597f6b9f6-blz6m\" (UID: \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\") " pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:31:46.965107 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:46.964951    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-metrics-certs\") pod \"maas-controller-5597f6b9f6-blz6m\" (UID: \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\") " pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:31:47.065889 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:47.065853    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-webhook-cert\") pod \"maas-controller-5597f6b9f6-blz6m\" (UID: \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\") " pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:31:47.066113 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:47.065898    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbdp\" (UniqueName: \"kubernetes.io/projected/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-kube-api-access-wnbdp\") pod \"maas-controller-5597f6b9f6-blz6m\" (UID: \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\") " pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:31:47.066113 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:47.065941    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-metrics-certs\") pod \"maas-controller-5597f6b9f6-blz6m\" (UID: \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\") " pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:31:47.068530 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:47.068503    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-webhook-cert\") pod \"maas-controller-5597f6b9f6-blz6m\" (UID: \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\") " pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:31:47.068627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:47.068558    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-metrics-certs\") pod \"maas-controller-5597f6b9f6-blz6m\" (UID: \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\") " pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:31:47.091234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:47.091169    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbdp\" (UniqueName: \"kubernetes.io/projected/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-kube-api-access-wnbdp\") pod \"maas-controller-5597f6b9f6-blz6m\" (UID: \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\") " pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:31:47.147272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:47.147237    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:31:47.195141 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:47.195107    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-689ccbb994-jcbnx" event={"ID":"49a53aee-e53f-41f8-97eb-182cc35f94c2","Type":"ContainerStarted","Data":"34a3937422bf0e58805401cfc9ff81b4e1f546ab4f2dcf0c61c20270fbcaf520"}
Nov 02 12:31:47.449494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:47.449464    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-blz6m"]
Nov 02 12:31:47.452858 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:31:47.452830    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d5d9f7_939e_4c5e_8785_921c13b4ebf4.slice/crio-22aa186183c8b54870dd1fce9ae4e7d3c48951b7469880dac06b17672a67aa30 WatchSource:0}: Error finding container 22aa186183c8b54870dd1fce9ae4e7d3c48951b7469880dac06b17672a67aa30: Status 404 returned error can't find the container with id 22aa186183c8b54870dd1fce9ae4e7d3c48951b7469880dac06b17672a67aa30
Nov 02 12:31:48.204824 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:48.204783    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5597f6b9f6-blz6m" event={"ID":"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4","Type":"ContainerStarted","Data":"22aa186183c8b54870dd1fce9ae4e7d3c48951b7469880dac06b17672a67aa30"}
Nov 02 12:31:50.099656 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:50.099577    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:50.217830 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:50.217797    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-689ccbb994-jcbnx" event={"ID":"49a53aee-e53f-41f8-97eb-182cc35f94c2","Type":"ContainerStarted","Data":"50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e"}
Nov 02 12:31:50.218044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:50.217923    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:31:50.234574 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:50.234545    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:50.264608 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:50.264557    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-689ccbb994-jcbnx" podStartSLOduration=1.425984058 podStartE2EDuration="4.264539566s" totalImagesPullingTime="2.838555508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:31:50.264300419 +0000 UTC m=+256.371003271" watchObservedRunningTime="2026-11-02 12:31:50.264539566 +0000 UTC m=+256.371242419"
Nov 02 12:31:51.223654 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.223608    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5597f6b9f6-blz6m" event={"ID":"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4","Type":"ContainerStarted","Data":"b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025"}
Nov 02 12:31:51.224130 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.223701    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:31:51.275147 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.275092    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5597f6b9f6-blz6m" podStartSLOduration=2.507502064 podStartE2EDuration="5.27507797s" totalImagesPullingTime="2.767575906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:31:51.273627892 +0000 UTC m=+257.380330742" watchObservedRunningTime="2026-11-02 12:31:51.27507797 +0000 UTC m=+257.381780820"
Nov 02 12:31:51.632244 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.632210    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"]
Nov 02 12:31:51.635779 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.635758    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.657251 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.657224    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-hbpgq\""
Nov 02 12:31:51.658770 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.658746    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 02 12:31:51.661201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.661166    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"]
Nov 02 12:31:51.670961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.670934    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 02 12:31:51.671122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.671037    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 02 12:31:51.715619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.715580    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/637730bd-76f5-4bff-8dc2-ba86e4e5254d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cvvr4\" (UID: \"637730bd-76f5-4bff-8dc2-ba86e4e5254d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.715788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.715641    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/637730bd-76f5-4bff-8dc2-ba86e4e5254d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cvvr4\" (UID: \"637730bd-76f5-4bff-8dc2-ba86e4e5254d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.715841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.715814    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgnpd\" (UniqueName: \"kubernetes.io/projected/637730bd-76f5-4bff-8dc2-ba86e4e5254d-kube-api-access-wgnpd\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cvvr4\" (UID: \"637730bd-76f5-4bff-8dc2-ba86e4e5254d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.715883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.715865    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/637730bd-76f5-4bff-8dc2-ba86e4e5254d-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cvvr4\" (UID: \"637730bd-76f5-4bff-8dc2-ba86e4e5254d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.746295 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.746260    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:51.816510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.816478    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgnpd\" (UniqueName: \"kubernetes.io/projected/637730bd-76f5-4bff-8dc2-ba86e4e5254d-kube-api-access-wgnpd\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cvvr4\" (UID: \"637730bd-76f5-4bff-8dc2-ba86e4e5254d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.816510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.816513    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/637730bd-76f5-4bff-8dc2-ba86e4e5254d-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cvvr4\" (UID: \"637730bd-76f5-4bff-8dc2-ba86e4e5254d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.816727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.816557    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/637730bd-76f5-4bff-8dc2-ba86e4e5254d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cvvr4\" (UID: \"637730bd-76f5-4bff-8dc2-ba86e4e5254d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.816727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.816588    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/637730bd-76f5-4bff-8dc2-ba86e4e5254d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cvvr4\" (UID: \"637730bd-76f5-4bff-8dc2-ba86e4e5254d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.817462 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.817385    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/637730bd-76f5-4bff-8dc2-ba86e4e5254d-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cvvr4\" (UID: \"637730bd-76f5-4bff-8dc2-ba86e4e5254d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.817588 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.817569    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/637730bd-76f5-4bff-8dc2-ba86e4e5254d-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cvvr4\" (UID: \"637730bd-76f5-4bff-8dc2-ba86e4e5254d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.819172 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.819152    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/637730bd-76f5-4bff-8dc2-ba86e4e5254d-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cvvr4\" (UID: \"637730bd-76f5-4bff-8dc2-ba86e4e5254d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.842691 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.842661    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgnpd\" (UniqueName: \"kubernetes.io/projected/637730bd-76f5-4bff-8dc2-ba86e4e5254d-kube-api-access-wgnpd\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cvvr4\" (UID: \"637730bd-76f5-4bff-8dc2-ba86e4e5254d\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:51.871884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.871849    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:31:51.949403 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:51.949326    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:31:52.252281 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:52.252252    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"]
Nov 02 12:31:52.253126 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:31:52.253094    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637730bd_76f5_4bff_8dc2_ba86e4e5254d.slice/crio-72a29e4d94fc2bab3c3552440fc69aeea1dbd492ba16ae51d67393fea5ae20a2 WatchSource:0}: Error finding container 72a29e4d94fc2bab3c3552440fc69aeea1dbd492ba16ae51d67393fea5ae20a2: Status 404 returned error can't find the container with id 72a29e4d94fc2bab3c3552440fc69aeea1dbd492ba16ae51d67393fea5ae20a2
Nov 02 12:31:53.233688 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:31:53.233646    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4" event={"ID":"637730bd-76f5-4bff-8dc2-ba86e4e5254d","Type":"ContainerStarted","Data":"72a29e4d94fc2bab3c3552440fc69aeea1dbd492ba16ae51d67393fea5ae20a2"}
Nov 02 12:32:00.430784 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.430737    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:00.576062 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.576024    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:00.751607 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.751516    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7js7d"]
Nov 02 12:32:00.762883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.762846    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:00.778553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.777906    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d"
Nov 02 12:32:00.788601 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.788518    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7js7d"]
Nov 02 12:32:00.804182 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.804144    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7pk9x\""
Nov 02 12:32:00.809423 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.809400    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 02 12:32:00.811422 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.811399    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"]
Nov 02 12:32:00.816092 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.816067    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"
Nov 02 12:32:00.830476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.830448    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f09ae13-0192-4a68-a935-39846f02caca-cert\") pod \"kserve-controller-manager-776cc66d47-7js7d\" (UID: \"2f09ae13-0192-4a68-a935-39846f02caca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d"
Nov 02 12:32:00.830755 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.830701    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4xvw\" (UniqueName: \"kubernetes.io/projected/2f09ae13-0192-4a68-a935-39846f02caca-kube-api-access-l4xvw\") pod \"kserve-controller-manager-776cc66d47-7js7d\" (UID: \"2f09ae13-0192-4a68-a935-39846f02caca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d"
Nov 02 12:32:00.834415 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.834350    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"]
Nov 02 12:32:00.850204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.850176    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nwfcr"]
Nov 02 12:32:00.854218 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.854198    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr"
Nov 02 12:32:00.880340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.880308    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nwfcr"]
Nov 02 12:32:00.904422 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.904392    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 02 12:32:00.904605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.904459    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-lhckf\""
Nov 02 12:32:00.904671 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.904600    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-r75fj\""
Nov 02 12:32:00.904730 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.904704    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 02 12:32:00.920721 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.920694    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-vwd9x"]
Nov 02 12:32:00.924621 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.924600    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:00.932027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.931518    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39c122b3-3586-4101-b48b-30517b4955d0-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-n5tzn\" (UID: \"39c122b3-3586-4101-b48b-30517b4955d0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"
Nov 02 12:32:00.932027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.931591    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4xvw\" (UniqueName: \"kubernetes.io/projected/2f09ae13-0192-4a68-a935-39846f02caca-kube-api-access-l4xvw\") pod \"kserve-controller-manager-776cc66d47-7js7d\" (UID: \"2f09ae13-0192-4a68-a935-39846f02caca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d"
Nov 02 12:32:00.932027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.931724    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8891b09c-a283-422f-a4ea-25f84ea846a8-tls-certs\") pod \"model-serving-api-75bd89fdfc-nwfcr\" (UID: \"8891b09c-a283-422f-a4ea-25f84ea846a8\") " pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr"
Nov 02 12:32:00.932027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.931758    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tkll\" (UniqueName: \"kubernetes.io/projected/8891b09c-a283-422f-a4ea-25f84ea846a8-kube-api-access-8tkll\") pod \"model-serving-api-75bd89fdfc-nwfcr\" (UID: \"8891b09c-a283-422f-a4ea-25f84ea846a8\") " pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr"
Nov 02 12:32:00.932027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.931824    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtw9\" (UniqueName: \"kubernetes.io/projected/39c122b3-3586-4101-b48b-30517b4955d0-kube-api-access-mhtw9\") pod \"llmisvc-controller-manager-869d5dcc5d-n5tzn\" (UID: \"39c122b3-3586-4101-b48b-30517b4955d0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"
Nov 02 12:32:00.932027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.931866    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f09ae13-0192-4a68-a935-39846f02caca-cert\") pod \"kserve-controller-manager-776cc66d47-7js7d\" (UID: \"2f09ae13-0192-4a68-a935-39846f02caca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d"
Nov 02 12:32:00.935075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.935049    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f09ae13-0192-4a68-a935-39846f02caca-cert\") pod \"kserve-controller-manager-776cc66d47-7js7d\" (UID: \"2f09ae13-0192-4a68-a935-39846f02caca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d"
Nov 02 12:32:00.935277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.935251    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:00.970438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.970408    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-vwd9x"]
Nov 02 12:32:00.978064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:00.978033    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4xvw\" (UniqueName: \"kubernetes.io/projected/2f09ae13-0192-4a68-a935-39846f02caca-kube-api-access-l4xvw\") pod \"kserve-controller-manager-776cc66d47-7js7d\" (UID: \"2f09ae13-0192-4a68-a935-39846f02caca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d"
Nov 02 12:32:01.004280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.004094    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 02 12:32:01.004280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.004206    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 02 12:32:01.004920 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.004894    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-lsmmk\""
Nov 02 12:32:01.032731 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.032697    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b097954-45fc-45e8-937e-4ece9fbdc1c6-cert\") pod \"odh-model-controller-769875db56-vwd9x\" (UID: \"8b097954-45fc-45e8-937e-4ece9fbdc1c6\") " pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:01.032952 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.032761    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8891b09c-a283-422f-a4ea-25f84ea846a8-tls-certs\") pod \"model-serving-api-75bd89fdfc-nwfcr\" (UID: \"8891b09c-a283-422f-a4ea-25f84ea846a8\") " pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr"
Nov 02 12:32:01.032952 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.032843    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tkll\" (UniqueName: \"kubernetes.io/projected/8891b09c-a283-422f-a4ea-25f84ea846a8-kube-api-access-8tkll\") pod \"model-serving-api-75bd89fdfc-nwfcr\" (UID: \"8891b09c-a283-422f-a4ea-25f84ea846a8\") " pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr"
Nov 02 12:32:01.032952 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.032898    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b097954-45fc-45e8-937e-4ece9fbdc1c6-metrics-certs\") pod \"odh-model-controller-769875db56-vwd9x\" (UID: \"8b097954-45fc-45e8-937e-4ece9fbdc1c6\") " pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:01.033217 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.032970    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtw9\" (UniqueName: \"kubernetes.io/projected/39c122b3-3586-4101-b48b-30517b4955d0-kube-api-access-mhtw9\") pod \"llmisvc-controller-manager-869d5dcc5d-n5tzn\" (UID: \"39c122b3-3586-4101-b48b-30517b4955d0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"
Nov 02 12:32:01.033217 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.033048    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62ckt\" (UniqueName: \"kubernetes.io/projected/8b097954-45fc-45e8-937e-4ece9fbdc1c6-kube-api-access-62ckt\") pod \"odh-model-controller-769875db56-vwd9x\" (UID: \"8b097954-45fc-45e8-937e-4ece9fbdc1c6\") " pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:01.033217 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.033085    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39c122b3-3586-4101-b48b-30517b4955d0-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-n5tzn\" (UID: \"39c122b3-3586-4101-b48b-30517b4955d0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"
Nov 02 12:32:01.035869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.035842    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8891b09c-a283-422f-a4ea-25f84ea846a8-tls-certs\") pod \"model-serving-api-75bd89fdfc-nwfcr\" (UID: \"8891b09c-a283-422f-a4ea-25f84ea846a8\") " pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr"
Nov 02 12:32:01.036020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.035918    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39c122b3-3586-4101-b48b-30517b4955d0-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-n5tzn\" (UID: \"39c122b3-3586-4101-b48b-30517b4955d0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"
Nov 02 12:32:01.060515 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.060482    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tkll\" (UniqueName: \"kubernetes.io/projected/8891b09c-a283-422f-a4ea-25f84ea846a8-kube-api-access-8tkll\") pod \"model-serving-api-75bd89fdfc-nwfcr\" (UID: \"8891b09c-a283-422f-a4ea-25f84ea846a8\") " pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr"
Nov 02 12:32:01.060747 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.060725    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtw9\" (UniqueName: \"kubernetes.io/projected/39c122b3-3586-4101-b48b-30517b4955d0-kube-api-access-mhtw9\") pod \"llmisvc-controller-manager-869d5dcc5d-n5tzn\" (UID: \"39c122b3-3586-4101-b48b-30517b4955d0\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"
Nov 02 12:32:01.095507 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.095475    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d"
Nov 02 12:32:01.131270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.131020    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"
Nov 02 12:32:01.134429 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.134401    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b097954-45fc-45e8-937e-4ece9fbdc1c6-metrics-certs\") pod \"odh-model-controller-769875db56-vwd9x\" (UID: \"8b097954-45fc-45e8-937e-4ece9fbdc1c6\") " pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:01.134574 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.134486    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62ckt\" (UniqueName: \"kubernetes.io/projected/8b097954-45fc-45e8-937e-4ece9fbdc1c6-kube-api-access-62ckt\") pod \"odh-model-controller-769875db56-vwd9x\" (UID: \"8b097954-45fc-45e8-937e-4ece9fbdc1c6\") " pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:01.134637 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.134595    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b097954-45fc-45e8-937e-4ece9fbdc1c6-cert\") pod \"odh-model-controller-769875db56-vwd9x\" (UID: \"8b097954-45fc-45e8-937e-4ece9fbdc1c6\") " pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:01.137146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.137118    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b097954-45fc-45e8-937e-4ece9fbdc1c6-cert\") pod \"odh-model-controller-769875db56-vwd9x\" (UID: \"8b097954-45fc-45e8-937e-4ece9fbdc1c6\") " pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:01.137536 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.137512    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b097954-45fc-45e8-937e-4ece9fbdc1c6-metrics-certs\") pod \"odh-model-controller-769875db56-vwd9x\" (UID: \"8b097954-45fc-45e8-937e-4ece9fbdc1c6\") " pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:01.165920 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.165864    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62ckt\" (UniqueName: \"kubernetes.io/projected/8b097954-45fc-45e8-937e-4ece9fbdc1c6-kube-api-access-62ckt\") pod \"odh-model-controller-769875db56-vwd9x\" (UID: \"8b097954-45fc-45e8-937e-4ece9fbdc1c6\") " pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:01.170871 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.170197    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr"
Nov 02 12:32:01.227139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.226288    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:32:01.241122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.240808    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:01.360236 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.360148    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:01.409018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.408961    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-7js7d"]
Nov 02 12:32:01.410251 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:32:01.410209    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f09ae13_0192_4a68_a935_39846f02caca.slice/crio-9db280b3aff62dfb100ffceee9b262c28c6bac8261afd8ff9b08c83e3ac1e830 WatchSource:0}: Error finding container 9db280b3aff62dfb100ffceee9b262c28c6bac8261afd8ff9b08c83e3ac1e830: Status 404 returned error can't find the container with id 9db280b3aff62dfb100ffceee9b262c28c6bac8261afd8ff9b08c83e3ac1e830
Nov 02 12:32:01.464536 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.464504    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"]
Nov 02 12:32:01.465028 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:32:01.464973    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod39c122b3_3586_4101_b48b_30517b4955d0.slice/crio-ff68c71d23e3bc03fe9c600f0dbcecaf695b69be8e527945acdfa305d6786ecf WatchSource:0}: Error finding container ff68c71d23e3bc03fe9c600f0dbcecaf695b69be8e527945acdfa305d6786ecf: Status 404 returned error can't find the container with id ff68c71d23e3bc03fe9c600f0dbcecaf695b69be8e527945acdfa305d6786ecf
Nov 02 12:32:01.510388 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:32:01.510318    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8891b09c_a283_422f_a4ea_25f84ea846a8.slice/crio-56361f868b148890614c687097cd0101d58e534e6d4892098a241e7f0037e1ca WatchSource:0}: Error finding container 56361f868b148890614c687097cd0101d58e534e6d4892098a241e7f0037e1ca: Status 404 returned error can't find the container with id 56361f868b148890614c687097cd0101d58e534e6d4892098a241e7f0037e1ca
Nov 02 12:32:01.510658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.510631    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nwfcr"]
Nov 02 12:32:01.567618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:01.567588    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-vwd9x"]
Nov 02 12:32:01.568820 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:32:01.568785    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b097954_45fc_45e8_937e_4ece9fbdc1c6.slice/crio-979d6f0c397476aa0ea9a0e990512d3420afa34c298a55871534d8ae7185b959 WatchSource:0}: Error finding container 979d6f0c397476aa0ea9a0e990512d3420afa34c298a55871534d8ae7185b959: Status 404 returned error can't find the container with id 979d6f0c397476aa0ea9a0e990512d3420afa34c298a55871534d8ae7185b959
Nov 02 12:32:02.229928 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.229898    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:32:02.293495 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.293403    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn" event={"ID":"39c122b3-3586-4101-b48b-30517b4955d0","Type":"ContainerStarted","Data":"ff68c71d23e3bc03fe9c600f0dbcecaf695b69be8e527945acdfa305d6786ecf"}
Nov 02 12:32:02.300586 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.300506    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr" event={"ID":"8891b09c-a283-422f-a4ea-25f84ea846a8","Type":"ContainerStarted","Data":"56361f868b148890614c687097cd0101d58e534e6d4892098a241e7f0037e1ca"}
Nov 02 12:32:02.302764 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.302690    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-vwd9x" event={"ID":"8b097954-45fc-45e8-937e-4ece9fbdc1c6","Type":"ContainerStarted","Data":"979d6f0c397476aa0ea9a0e990512d3420afa34c298a55871534d8ae7185b959"}
Nov 02 12:32:02.304972 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.304886    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d" event={"ID":"2f09ae13-0192-4a68-a935-39846f02caca","Type":"ContainerStarted","Data":"9db280b3aff62dfb100ffceee9b262c28c6bac8261afd8ff9b08c83e3ac1e830"}
Nov 02 12:32:02.309260 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.309236    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-689ccbb994-jcbnx"]
Nov 02 12:32:02.310516 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.310128    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-689ccbb994-jcbnx" podUID="49a53aee-e53f-41f8-97eb-182cc35f94c2" containerName="manager" containerID="cri-o://50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e" gracePeriod=30
Nov 02 12:32:02.662686 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.662651    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:32:02.753551 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.752706    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49a53aee-e53f-41f8-97eb-182cc35f94c2-kube-api-access-7x742\" (UniqueName: \"kubernetes.io/projected/49a53aee-e53f-41f8-97eb-182cc35f94c2-kube-api-access-7x742\") pod \"49a53aee-e53f-41f8-97eb-182cc35f94c2\" (UID: \"49a53aee-e53f-41f8-97eb-182cc35f94c2\") "
Nov 02 12:32:02.753551 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.752813    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-webhook-cert\" (UniqueName: \"kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-webhook-cert\") pod \"49a53aee-e53f-41f8-97eb-182cc35f94c2\" (UID: \"49a53aee-e53f-41f8-97eb-182cc35f94c2\") "
Nov 02 12:32:02.753551 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.752864    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-metrics-certs\") pod \"49a53aee-e53f-41f8-97eb-182cc35f94c2\" (UID: \"49a53aee-e53f-41f8-97eb-182cc35f94c2\") "
Nov 02 12:32:02.760106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.758076    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-webhook-cert" pod "49a53aee-e53f-41f8-97eb-182cc35f94c2" (UID: "49a53aee-e53f-41f8-97eb-182cc35f94c2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:32:02.772643 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.772608    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-metrics-certs" pod "49a53aee-e53f-41f8-97eb-182cc35f94c2" (UID: "49a53aee-e53f-41f8-97eb-182cc35f94c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:32:02.774683 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.774651    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a53aee-e53f-41f8-97eb-182cc35f94c2-kube-api-access-7x742" pod "49a53aee-e53f-41f8-97eb-182cc35f94c2" (UID: "49a53aee-e53f-41f8-97eb-182cc35f94c2"). InnerVolumeSpecName "kube-api-access-7x742". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:32:02.854254 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.854161    5879 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-webhook-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:32:02.854254 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.854198    5879 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49a53aee-e53f-41f8-97eb-182cc35f94c2-metrics-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:32:02.854254 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:02.854209    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x742\" (UniqueName: \"kubernetes.io/projected/49a53aee-e53f-41f8-97eb-182cc35f94c2-kube-api-access-7x742\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:32:03.317201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.314199    5879 generic.go:356] "Generic (PLEG): container finished" podID="49a53aee-e53f-41f8-97eb-182cc35f94c2" containerID="50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e" exitCode=0
Nov 02 12:32:03.317201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.314350    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-689ccbb994-jcbnx" event={"ID":"49a53aee-e53f-41f8-97eb-182cc35f94c2","Type":"ContainerDied","Data":"50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e"}
Nov 02 12:32:03.317201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.314401    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-689ccbb994-jcbnx" event={"ID":"49a53aee-e53f-41f8-97eb-182cc35f94c2","Type":"ContainerDied","Data":"34a3937422bf0e58805401cfc9ff81b4e1f546ab4f2dcf0c61c20270fbcaf520"}
Nov 02 12:32:03.317201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.314430    5879 scope.go:122] "RemoveContainer" containerID="50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e"
Nov 02 12:32:03.317201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.314592    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-689ccbb994-jcbnx"
Nov 02 12:32:03.338859 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.338835    5879 scope.go:122] "RemoveContainer" containerID="50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e"
Nov 02 12:32:03.339314 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:32:03.339186    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e\": container with ID starting with 50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e not found: ID does not exist" containerID="50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e"
Nov 02 12:32:03.339314 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.339226    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e"} err="failed to get container status \"50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e\": rpc error: code = NotFound desc = could not find container \"50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e\": container with ID starting with 50cd58ee249597881424682c50e0c528995fcc72d50d1e3b75ad1e912e0d7f3e not found: ID does not exist"
Nov 02 12:32:03.374070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.373986    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-689ccbb994-jcbnx"]
Nov 02 12:32:03.377665 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.377628    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:03.388255 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.388191    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-689ccbb994-jcbnx"]
Nov 02 12:32:03.502767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.502682    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:03.623784 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.623696    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:03.746138 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.746065    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:03.870967 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:03.870882    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:04.019103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:04.019024    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49a53aee-e53f-41f8-97eb-182cc35f94c2" path="/var/lib/kubelet/pods/49a53aee-e53f-41f8-97eb-182cc35f94c2/volumes"
Nov 02 12:32:05.330620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:05.330584    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d" event={"ID":"2f09ae13-0192-4a68-a935-39846f02caca","Type":"ContainerStarted","Data":"51652a1fc5719f743abe082e87296f33c50487ee60f1e760afe29d3a35750fc7"}
Nov 02 12:32:05.331037 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:05.330884    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d"
Nov 02 12:32:05.332808 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:05.332786    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn" event={"ID":"39c122b3-3586-4101-b48b-30517b4955d0","Type":"ContainerStarted","Data":"5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012"}
Nov 02 12:32:05.334791 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:05.334768    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr" event={"ID":"8891b09c-a283-422f-a4ea-25f84ea846a8","Type":"ContainerStarted","Data":"c2d160c3f97971e2dc5bd202ded3cd9336705d0c38d01edb7475322e30394239"}
Nov 02 12:32:05.334941 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:05.334918    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr"
Nov 02 12:32:05.376348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:05.376290    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d" podStartSLOduration=1.794341188 podStartE2EDuration="5.376273176s" totalImagesPullingTime="3.581931988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:05.373883207 +0000 UTC m=+271.480586058" watchObservedRunningTime="2026-11-02 12:32:05.376273176 +0000 UTC m=+271.482976026"
Nov 02 12:32:05.418942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:05.418505    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr" podStartSLOduration=2.273182575 podStartE2EDuration="5.41848602s" totalImagesPullingTime="3.145303445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:05.41779526 +0000 UTC m=+271.524498111" watchObservedRunningTime="2026-11-02 12:32:05.41848602 +0000 UTC m=+271.525188865"
Nov 02 12:32:06.341669 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:06.341631    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4" event={"ID":"637730bd-76f5-4bff-8dc2-ba86e4e5254d","Type":"ContainerStarted","Data":"b0755a8476d4f231165fdfd5d7c8cab00b487b27ff92cf1ba6a563f7f6ec48a7"}
Nov 02 12:32:06.342208 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:06.341733    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:32:06.343472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:06.343347    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-vwd9x" event={"ID":"8b097954-45fc-45e8-937e-4ece9fbdc1c6","Type":"ContainerStarted","Data":"398de69f8e28f0111976c8a20fefb5d92722dcc6381e5d54aaece39e0dfcb9b4"}
Nov 02 12:32:06.343472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:06.343414    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:06.387214 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:06.387151    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4" podStartSLOduration=1.421005775 podStartE2EDuration="15.387132748s" totalImagesPullingTime="13.966126973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:06.385973804 +0000 UTC m=+272.492676654" watchObservedRunningTime="2026-11-02 12:32:06.387132748 +0000 UTC m=+272.493835602"
Nov 02 12:32:06.435550 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:06.435495    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn" podStartSLOduration=2.743259737 podStartE2EDuration="6.435482949s" totalImagesPullingTime="3.692223212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:06.433801057 +0000 UTC m=+272.540503908" watchObservedRunningTime="2026-11-02 12:32:06.435482949 +0000 UTC m=+272.542185778"
Nov 02 12:32:06.478813 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:06.478714    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-vwd9x" podStartSLOduration=1.843004205 podStartE2EDuration="6.478697402s" totalImagesPullingTime="4.635693197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:32:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:06.477499734 +0000 UTC m=+272.584202587" watchObservedRunningTime="2026-11-02 12:32:06.478697402 +0000 UTC m=+272.585400293"
Nov 02 12:32:11.343839 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:11.343804    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"
Nov 02 12:32:14.820050 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:32:14.819954    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:32:16.347397 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:16.347370    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-nwfcr"
Nov 02 12:32:17.350063 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:17.350039    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-vwd9x"
Nov 02 12:32:17.353289 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:17.353269    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cvvr4"
Nov 02 12:32:20.233659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:20.233618    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:20.353840 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:20.353796    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:21.167244 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:21.167206    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:21.297389 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:21.297347    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:22.534659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:22.534627    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:22.746739 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:22.746708    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:22.861014 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:22.860915    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:32.706189 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:32.706146    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:32.825156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:32.825119    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:32.974427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:32.974360    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4s8hv"]
Nov 02 12:32:32.974855 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:32.974833    5879 cpu_manager.go:414] "containerMap: removing container" podUID="49a53aee-e53f-41f8-97eb-182cc35f94c2" containerName="manager"
Nov 02 12:32:32.974855 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:32.974852    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49a53aee-e53f-41f8-97eb-182cc35f94c2" containerName="manager"
Nov 02 12:32:32.975102 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:32.974944    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="49a53aee-e53f-41f8-97eb-182cc35f94c2" containerName="manager"
Nov 02 12:32:33.113720 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.113687    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:32:33.205343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.205292    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4s8hv"]
Nov 02 12:32:33.205541 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.205354    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"]
Nov 02 12:32:33.205541 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.205390    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4s8hv"
Nov 02 12:32:33.226091 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.226015    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"]
Nov 02 12:32:33.226231 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.226090    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.235569 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.235548    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x7vbt\""
Nov 02 12:32:33.283352 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.283324    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qjrt\" (UniqueName: \"kubernetes.io/projected/942b6c64-2ce3-4780-820f-b56b4ad86621-kube-api-access-5qjrt\") pod \"authorino-6f6944bcdd-4s8hv\" (UID: \"942b6c64-2ce3-4780-820f-b56b4ad86621\") " pod="kuadrant-system/authorino-6f6944bcdd-4s8hv"
Nov 02 12:32:33.283352 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.283353    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-maas-api-tls\") pod \"maas-api-65f6775b5-8jg8r\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.283568 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.283447    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhjp\" (UniqueName: \"kubernetes.io/projected/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-kube-api-access-wwhjp\") pod \"maas-api-65f6775b5-8jg8r\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.283568 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.283502    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-openshift-service-ca\") pod \"maas-api-65f6775b5-8jg8r\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.283722 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.283708    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-metrics-certs\") pod \"maas-api-65f6775b5-8jg8r\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.309272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.309241    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4s8hv"]
Nov 02 12:32:33.309474 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:32:33.309455    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-5qjrt], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-4s8hv" podUID="942b6c64-2ce3-4780-820f-b56b4ad86621"
Nov 02 12:32:33.336356 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.336327    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 12:32:33.336356 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.336344    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 12:32:33.336602 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.336434    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-sr5s9\""
Nov 02 12:32:33.384380 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.384351    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-openshift-service-ca\") pod \"maas-api-65f6775b5-8jg8r\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.384547 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.384399    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-metrics-certs\") pod \"maas-api-65f6775b5-8jg8r\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.384547 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.384443    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjrt\" (UniqueName: \"kubernetes.io/projected/942b6c64-2ce3-4780-820f-b56b4ad86621-kube-api-access-5qjrt\") pod \"authorino-6f6944bcdd-4s8hv\" (UID: \"942b6c64-2ce3-4780-820f-b56b4ad86621\") " pod="kuadrant-system/authorino-6f6944bcdd-4s8hv"
Nov 02 12:32:33.384547 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.384456    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-maas-api-tls\") pod \"maas-api-65f6775b5-8jg8r\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.384547 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.384497    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhjp\" (UniqueName: \"kubernetes.io/projected/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-kube-api-access-wwhjp\") pod \"maas-api-65f6775b5-8jg8r\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.385094 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.385071    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-openshift-service-ca\") pod \"maas-api-65f6775b5-8jg8r\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.387053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.387033    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-metrics-certs\") pod \"maas-api-65f6775b5-8jg8r\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.387134 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.387122    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-maas-api-tls\") pod \"maas-api-65f6775b5-8jg8r\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.414713 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.414685    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhjp\" (UniqueName: \"kubernetes.io/projected/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-kube-api-access-wwhjp\") pod \"maas-api-65f6775b5-8jg8r\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") " pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.417531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.417504    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjrt\" (UniqueName: \"kubernetes.io/projected/942b6c64-2ce3-4780-820f-b56b4ad86621-kube-api-access-5qjrt\") pod \"authorino-6f6944bcdd-4s8hv\" (UID: \"942b6c64-2ce3-4780-820f-b56b4ad86621\") " pod="kuadrant-system/authorino-6f6944bcdd-4s8hv"
Nov 02 12:32:33.466721 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.466686    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4s8hv"
Nov 02 12:32:33.516163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.516078    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4s8hv"
Nov 02 12:32:33.558808 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.558772    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:33.586350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.586323    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942b6c64-2ce3-4780-820f-b56b4ad86621-kube-api-access-5qjrt\" (UniqueName: \"kubernetes.io/projected/942b6c64-2ce3-4780-820f-b56b4ad86621-kube-api-access-5qjrt\") pod \"942b6c64-2ce3-4780-820f-b56b4ad86621\" (UID: \"942b6c64-2ce3-4780-820f-b56b4ad86621\") "
Nov 02 12:32:33.588569 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.588542    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942b6c64-2ce3-4780-820f-b56b4ad86621-kube-api-access-5qjrt" pod "942b6c64-2ce3-4780-820f-b56b4ad86621" (UID: "942b6c64-2ce3-4780-820f-b56b4ad86621"). InnerVolumeSpecName "kube-api-access-5qjrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:32:33.687431 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.687395    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qjrt\" (UniqueName: \"kubernetes.io/projected/942b6c64-2ce3-4780-820f-b56b4ad86621-kube-api-access-5qjrt\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:32:33.873469 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:33.873439    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"]
Nov 02 12:32:33.874568 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:32:33.874533    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d1d69f_c28b_492f_8ffc_4ea1ed28cfce.slice/crio-fc5cdc98f5bc5d4405ede80b4c776e46ab42c052cdee60d6a8ba69767adc4b5f WatchSource:0}: Error finding container fc5cdc98f5bc5d4405ede80b4c776e46ab42c052cdee60d6a8ba69767adc4b5f: Status 404 returned error can't find the container with id fc5cdc98f5bc5d4405ede80b4c776e46ab42c052cdee60d6a8ba69767adc4b5f
Nov 02 12:32:34.473926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:34.473883    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r" event={"ID":"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce","Type":"ContainerStarted","Data":"fc5cdc98f5bc5d4405ede80b4c776e46ab42c052cdee60d6a8ba69767adc4b5f"}
Nov 02 12:32:34.474139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:34.473968    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-4s8hv"
Nov 02 12:32:34.555469 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:34.555429    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4s8hv"]
Nov 02 12:32:34.558595 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:34.558553    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-4s8hv"]
Nov 02 12:32:36.016565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:36.016487    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942b6c64-2ce3-4780-820f-b56b4ad86621" path="/var/lib/kubelet/pods/942b6c64-2ce3-4780-820f-b56b4ad86621/volumes"
Nov 02 12:32:36.346359 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:36.346277    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"
Nov 02 12:32:36.349165 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:36.349141    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-7js7d"
Nov 02 12:32:36.486686 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:36.486650    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r" event={"ID":"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce","Type":"ContainerStarted","Data":"cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3"}
Nov 02 12:32:36.486856 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:36.486786    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:36.533032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:36.532933    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r" podStartSLOduration=1.649162295 podStartE2EDuration="3.532914577s" totalImagesPullingTime="1.883752282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:32:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:36.532595399 +0000 UTC m=+302.639298251" watchObservedRunningTime="2026-11-02 12:32:36.532914577 +0000 UTC m=+302.639617429"
Nov 02 12:32:42.497048 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:42.497018    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 12:32:44.820734 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:32:44.820695    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:32:56.431142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:56.431105    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"]
Nov 02 12:32:56.433900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:56.433882    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"
Nov 02 12:32:56.466285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:56.466257    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"]
Nov 02 12:32:56.613286 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:56.613250    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95zrq\" (UniqueName: \"kubernetes.io/projected/8e9e3acc-7a16-4166-bf7d-22cc70517482-kube-api-access-95zrq\") pod \"llmisvc-controller-manager-6466c554d8-vd4d6\" (UID: \"8e9e3acc-7a16-4166-bf7d-22cc70517482\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"
Nov 02 12:32:56.613472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:56.613310    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e9e3acc-7a16-4166-bf7d-22cc70517482-cert\") pod \"llmisvc-controller-manager-6466c554d8-vd4d6\" (UID: \"8e9e3acc-7a16-4166-bf7d-22cc70517482\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"
Nov 02 12:32:56.714969 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:56.714875    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95zrq\" (UniqueName: \"kubernetes.io/projected/8e9e3acc-7a16-4166-bf7d-22cc70517482-kube-api-access-95zrq\") pod \"llmisvc-controller-manager-6466c554d8-vd4d6\" (UID: \"8e9e3acc-7a16-4166-bf7d-22cc70517482\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"
Nov 02 12:32:56.714969 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:56.714932    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e9e3acc-7a16-4166-bf7d-22cc70517482-cert\") pod \"llmisvc-controller-manager-6466c554d8-vd4d6\" (UID: \"8e9e3acc-7a16-4166-bf7d-22cc70517482\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"
Nov 02 12:32:56.717535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:56.717513    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e9e3acc-7a16-4166-bf7d-22cc70517482-cert\") pod \"llmisvc-controller-manager-6466c554d8-vd4d6\" (UID: \"8e9e3acc-7a16-4166-bf7d-22cc70517482\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"
Nov 02 12:32:56.740263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:56.740240    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95zrq\" (UniqueName: \"kubernetes.io/projected/8e9e3acc-7a16-4166-bf7d-22cc70517482-kube-api-access-95zrq\") pod \"llmisvc-controller-manager-6466c554d8-vd4d6\" (UID: \"8e9e3acc-7a16-4166-bf7d-22cc70517482\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"
Nov 02 12:32:56.746116 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:56.746099    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"
Nov 02 12:32:57.043226 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:57.043199    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"]
Nov 02 12:32:57.043914 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:32:57.043890    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9e3acc_7a16_4166_bf7d_22cc70517482.slice/crio-bf6c719d950b4e2cbf53af9c2d85466640fe028dd6231e1385540e43ba31d691 WatchSource:0}: Error finding container bf6c719d950b4e2cbf53af9c2d85466640fe028dd6231e1385540e43ba31d691: Status 404 returned error can't find the container with id bf6c719d950b4e2cbf53af9c2d85466640fe028dd6231e1385540e43ba31d691
Nov 02 12:32:57.587599 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:57.587562    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6" event={"ID":"8e9e3acc-7a16-4166-bf7d-22cc70517482","Type":"ContainerStarted","Data":"8fbac65f9c5c835a979827b27efb5c40a3b576a27718aa878a018ef30f3e6cba"}
Nov 02 12:32:57.588016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:57.587622    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6" event={"ID":"8e9e3acc-7a16-4166-bf7d-22cc70517482","Type":"ContainerStarted","Data":"bf6c719d950b4e2cbf53af9c2d85466640fe028dd6231e1385540e43ba31d691"}
Nov 02 12:32:57.588016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:57.587716    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"
Nov 02 12:32:57.632785 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:32:57.632737    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6" podStartSLOduration=1.315691569 podStartE2EDuration="1.632721094s" totalImagesPullingTime="317.029525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:32:57.632585878 +0000 UTC m=+323.739288731" watchObservedRunningTime="2026-11-02 12:32:57.632721094 +0000 UTC m=+323.739423974"
Nov 02 12:33:14.819450 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:33:14.819414    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:33:28.595874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:28.595838    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vd4d6"
Nov 02 12:33:28.669300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:28.669265    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"]
Nov 02 12:33:28.669533 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:28.669496    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn" podUID="39c122b3-3586-4101-b48b-30517b4955d0" containerName="manager" containerID="cri-o://5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012" gracePeriod=30
Nov 02 12:33:28.922759 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:28.922736    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"
Nov 02 12:33:29.014491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.014453    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39c122b3-3586-4101-b48b-30517b4955d0-cert\" (UniqueName: \"kubernetes.io/secret/39c122b3-3586-4101-b48b-30517b4955d0-cert\") pod \"39c122b3-3586-4101-b48b-30517b4955d0\" (UID: \"39c122b3-3586-4101-b48b-30517b4955d0\") "
Nov 02 12:33:29.014682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.014519    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c122b3-3586-4101-b48b-30517b4955d0-kube-api-access-mhtw9\" (UniqueName: \"kubernetes.io/projected/39c122b3-3586-4101-b48b-30517b4955d0-kube-api-access-mhtw9\") pod \"39c122b3-3586-4101-b48b-30517b4955d0\" (UID: \"39c122b3-3586-4101-b48b-30517b4955d0\") "
Nov 02 12:33:29.016764 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.016712    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c122b3-3586-4101-b48b-30517b4955d0-kube-api-access-mhtw9" pod "39c122b3-3586-4101-b48b-30517b4955d0" (UID: "39c122b3-3586-4101-b48b-30517b4955d0"). InnerVolumeSpecName "kube-api-access-mhtw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:33:29.016899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.016730    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c122b3-3586-4101-b48b-30517b4955d0-cert" pod "39c122b3-3586-4101-b48b-30517b4955d0" (UID: "39c122b3-3586-4101-b48b-30517b4955d0"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:33:29.116234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.116195    5879 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/39c122b3-3586-4101-b48b-30517b4955d0-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:33:29.116234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.116227    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhtw9\" (UniqueName: \"kubernetes.io/projected/39c122b3-3586-4101-b48b-30517b4955d0-kube-api-access-mhtw9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:33:29.738909 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.738873    5879 generic.go:356] "Generic (PLEG): container finished" podID="39c122b3-3586-4101-b48b-30517b4955d0" containerID="5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012" exitCode=0
Nov 02 12:33:29.739374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.738927    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"
Nov 02 12:33:29.739374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.738966    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn" event={"ID":"39c122b3-3586-4101-b48b-30517b4955d0","Type":"ContainerDied","Data":"5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012"}
Nov 02 12:33:29.739374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.739042    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn" event={"ID":"39c122b3-3586-4101-b48b-30517b4955d0","Type":"ContainerDied","Data":"ff68c71d23e3bc03fe9c600f0dbcecaf695b69be8e527945acdfa305d6786ecf"}
Nov 02 12:33:29.739374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.739071    5879 scope.go:122] "RemoveContainer" containerID="5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012"
Nov 02 12:33:29.748386 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.748370    5879 scope.go:122] "RemoveContainer" containerID="5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012"
Nov 02 12:33:29.748668 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:33:29.748652    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012\": container with ID starting with 5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012 not found: ID does not exist" containerID="5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012"
Nov 02 12:33:29.748756 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.748680    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012"} err="failed to get container status \"5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012\": rpc error: code = NotFound desc = could not find container \"5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012\": container with ID starting with 5827ea556cee98a5fe5de818d2b0ef882df599de9cc2fe7d5fda31b6fcc61012 not found: ID does not exist"
Nov 02 12:33:29.795675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.795646    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"]
Nov 02 12:33:29.800121 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.800095    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-n5tzn"]
Nov 02 12:33:29.883587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:29.883554    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:33:30.015805 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:30.015732    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c122b3-3586-4101-b48b-30517b4955d0" path="/var/lib/kubelet/pods/39c122b3-3586-4101-b48b-30517b4955d0/volumes"
Nov 02 12:33:30.024536 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:30.024509    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:33:33.240170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:33.240133    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:33:33.944368 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:33.944334    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:33:34.066805 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:34.066776    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:33:44.821966 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:33:44.821889    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:33:59.322843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:59.322801    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:33:59.842494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:33:59.842455    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:01.104126 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:01.104093    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:01.246170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:01.246135    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:04.925006 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:04.924966    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:05.041776 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:05.041744    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:05.604676 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:05.604636    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:08.916301 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:08.916260    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:10.211789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:10.211752    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:12.910897 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:12.910838    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:14.383774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:14.383739    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:14.819203 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:34:14.819125    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:34:18.190400 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:18.190361    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:19.979041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:19.979004    5879 generic.go:356] "Generic (PLEG): container finished" podID="e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" containerID="b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025" exitCode=1
Nov 02 12:34:19.979418 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:19.979101    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5597f6b9f6-blz6m" event={"ID":"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4","Type":"ContainerDied","Data":"b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025"}
Nov 02 12:34:19.979553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:19.979538    5879 scope.go:122] "RemoveContainer" containerID="b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025"
Nov 02 12:34:20.800520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:20.800488    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:20.985767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:20.985733    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5597f6b9f6-blz6m" event={"ID":"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4","Type":"ContainerStarted","Data":"e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5"}
Nov 02 12:34:20.986178 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:20.985945    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:34:21.956717 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:21.956685    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:31.993952 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:31.993918    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 12:34:33.379380 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:33.379339    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:35.846590 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.846554    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"]
Nov 02 12:34:35.847181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.847160    5879 cpu_manager.go:414] "containerMap: removing container" podUID="39c122b3-3586-4101-b48b-30517b4955d0" containerName="manager"
Nov 02 12:34:35.847262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.847185    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c122b3-3586-4101-b48b-30517b4955d0" containerName="manager"
Nov 02 12:34:35.847358 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.847345    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c122b3-3586-4101-b48b-30517b4955d0" containerName="manager"
Nov 02 12:34:35.850286 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.850264    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.864859 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.864825    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.865076 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.864894    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.865076 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.864984    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.865076 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.865058    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.865282 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.865103    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.865282 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.865121    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v86p9\" (UniqueName: \"kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-kube-api-access-v86p9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.865282 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.865142    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.865282 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.865183    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.865282 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.865279    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.868136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.868115    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 12:34:35.891203 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.891166    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"]
Nov 02 12:34:35.964257 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.964222    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2nvrx\""
Nov 02 12:34:35.966660 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.966635    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.966728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.966683    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.966728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.966702    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.966728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.966722    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.966851 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.966766    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.966851 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.966791    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v86p9\" (UniqueName: \"kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-kube-api-access-v86p9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.966851 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.966821    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.967024 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.966926    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.967114 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.967095    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.967174 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.967099    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.967230 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.967198    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.967315 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.967291    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.967656 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.967636    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.967852 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.967828    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.969288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.969265    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:35.969519 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:35.969501    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:36.017549 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:36.017514    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:36.017702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:36.017652    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v86p9\" (UniqueName: \"kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-kube-api-access-v86p9\") pod \"maas-default-gateway-openshift-default-5bc884dd87-6bnj5\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:36.163226 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:36.163188    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:36.481354 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:36.481324    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"]
Nov 02 12:34:36.481980 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:34:36.481948    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5191e8a_315a_4d0b_bc4c_692ca33763bd.slice/crio-a28f658ceb3aedb75d60d7f821c208e792ff8181d6f91942bb91b770511d8a9e WatchSource:0}: Error finding container a28f658ceb3aedb75d60d7f821c208e792ff8181d6f91942bb91b770511d8a9e: Status 404 returned error can't find the container with id a28f658ceb3aedb75d60d7f821c208e792ff8181d6f91942bb91b770511d8a9e
Nov 02 12:34:36.484245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:36.484228    5879 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:34:37.066535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:37.066494    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" event={"ID":"c5191e8a-315a-4d0b-bc4c-692ca33763bd","Type":"ContainerStarted","Data":"a28f658ceb3aedb75d60d7f821c208e792ff8181d6f91942bb91b770511d8a9e"}
Nov 02 12:34:39.185976 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:39.185943    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:34:39.186299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:39.186059    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:34:39.186299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:39.186092    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:34:40.112202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:40.112153    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" event={"ID":"c5191e8a-315a-4d0b-bc4c-692ca33763bd","Type":"ContainerStarted","Data":"4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a"}
Nov 02 12:34:40.164059 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:40.164020    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:40.165203 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:40.165163    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-6bnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:34:40.165333 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:40.165228    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 12:34:40.206015 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:40.205122    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" podStartSLOduration=2.504497175 podStartE2EDuration="5.205099879s" totalImagesPullingTime="2.700602704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:40.203525787 +0000 UTC m=+426.310228648" watchObservedRunningTime="2026-11-02 12:34:40.205099879 +0000 UTC m=+426.311802730"
Nov 02 12:34:41.164019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:41.163955    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-6bnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:34:41.164195 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:41.164046    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 12:34:42.163836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:42.163800    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-6bnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:34:42.164290 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:42.163854    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 12:34:42.267924 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:42.267887    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:42.409351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:42.409305    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:43.164278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:43.164240    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-6bnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:34:43.164747 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:43.164299    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 12:34:43.954114 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:43.954078    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:44.107448 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:44.107406    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:34:44.164106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:44.164072    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-6bnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:34:44.164263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:44.164135    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 12:34:44.831447 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:34:44.831416    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:34:45.164573 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:45.164533    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-6bnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:34:45.164762 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:45.164603    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 12:34:46.164225 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:46.164189    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-6bnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 12:34:46.164630 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:46.164244    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 12:34:48.164389 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:48.164343    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-6bnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:34:48.164782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:48.164431    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:34:49.206227 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:49.206174    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-6bnj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:34:49.206781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:49.206243    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:34:49.526272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:49.526189    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:49.526625 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:49.526603    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:49.527258 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:49.527243    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:34:50.892774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:50.892742    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"]
Nov 02 12:34:50.899240 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:50.899218    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:50.924852 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:50.924815    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"]
Nov 02 12:34:51.032300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.032266    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.032300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.032299    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.032518 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.032321    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.032518 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.032397    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.032586 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.032518    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.032586 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.032577    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1ec3a45-ae38-4297-a894-27438a341dcf-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.032650 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.032636    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1ec3a45-ae38-4297-a894-27438a341dcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.032710 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.032696    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfpgb\" (UniqueName: \"kubernetes.io/projected/e1ec3a45-ae38-4297-a894-27438a341dcf-kube-api-access-wfpgb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.032744 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.032722    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1ec3a45-ae38-4297-a894-27438a341dcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.133710 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.133674    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfpgb\" (UniqueName: \"kubernetes.io/projected/e1ec3a45-ae38-4297-a894-27438a341dcf-kube-api-access-wfpgb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.133710 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.133712    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1ec3a45-ae38-4297-a894-27438a341dcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.133942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.133766    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.133942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.133782    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.133942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.133797    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.133942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.133827    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.133942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.133874    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.133942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.133933    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1ec3a45-ae38-4297-a894-27438a341dcf-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.134233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.133960    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1ec3a45-ae38-4297-a894-27438a341dcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.134355 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.134325    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.134500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.134379    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.134577 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.134543    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.134801 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.134549    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.134801 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.134596    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1ec3a45-ae38-4297-a894-27438a341dcf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.136202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.136182    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1ec3a45-ae38-4297-a894-27438a341dcf-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.136475 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.136456    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1ec3a45-ae38-4297-a894-27438a341dcf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.158909 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.158850    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1ec3a45-ae38-4297-a894-27438a341dcf-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.159018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.158930    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfpgb\" (UniqueName: \"kubernetes.io/projected/e1ec3a45-ae38-4297-a894-27438a341dcf-kube-api-access-wfpgb\") pod \"maas-default-gateway-openshift-default-5bc884dd87-m2dn7\" (UID: \"e1ec3a45-ae38-4297-a894-27438a341dcf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.214702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.214664    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:51.536503 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.536473    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"]
Nov 02 12:34:51.537811 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:34:51.537778    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ec3a45_ae38_4297_a894_27438a341dcf.slice/crio-cb763eaadf6e287c5b8264856defbf05f629524178f36589eac8534c5119195a WatchSource:0}: Error finding container cb763eaadf6e287c5b8264856defbf05f629524178f36589eac8534c5119195a: Status 404 returned error can't find the container with id cb763eaadf6e287c5b8264856defbf05f629524178f36589eac8534c5119195a
Nov 02 12:34:51.540689 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.540657    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:34:51.540763 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.540722    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:34:51.540763 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:51.540754    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:34:52.173438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:52.173402    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" event={"ID":"e1ec3a45-ae38-4297-a894-27438a341dcf","Type":"ContainerStarted","Data":"5f129a8b92258a368ef7b514ba4b224c813ea10ebe6dfa3869ad634917c0f4c7"}
Nov 02 12:34:52.173827 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:52.173484    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" event={"ID":"e1ec3a45-ae38-4297-a894-27438a341dcf","Type":"ContainerStarted","Data":"cb763eaadf6e287c5b8264856defbf05f629524178f36589eac8534c5119195a"}
Nov 02 12:34:52.215453 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:52.215418    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:52.216942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:52.216914    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:34:52.217082 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:52.216964    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 12:34:52.250969 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:52.250917    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podStartSLOduration=2.250905122 podStartE2EDuration="2.250905122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:34:52.249502806 +0000 UTC m=+438.356205658" watchObservedRunningTime="2026-11-02 12:34:52.250905122 +0000 UTC m=+438.357607974"
Nov 02 12:34:53.215500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:53.215452    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:34:53.215934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:53.215515    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 12:34:54.215624 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:54.215588    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:34:54.216151 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:54.215652    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 12:34:55.215657 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:55.215621    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:34:55.216108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:55.215704    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 12:34:56.215543 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:56.215517    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:34:56.215721 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:56.215564    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 12:34:57.215725 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:57.215686    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:34:57.216190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:57.215747    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 12:34:58.215301 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:58.215253    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 12:34:58.215493 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:58.215351    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 12:34:59.813139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:59.813106    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:59.813842 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:59.813806    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:34:59.814630 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:34:59.814610    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7"
Nov 02 12:35:14.819503 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:35:14.819416    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:35:22.161413 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:22.161341    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:22.302183 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:22.302153    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:22.433468 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:22.433391    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:22.673890 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:22.673853    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:22.835384 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:22.835270    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:22.991339 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:22.991303    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:23.123140 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:23.123108    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:23.282035 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:23.281982    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:23.485032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:23.484932    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:23.623336 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:23.623289    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:24.080882 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:24.080819    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:24.550749 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:24.550711    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:24.673857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:24.673803    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:25.287076 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:25.287041    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:25.409980 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:25.409941    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:26.028168 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:26.028132    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:26.154472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:26.154440    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:26.606351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:26.606323    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:26.734431 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:26.734390    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:26.871406 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:26.871378    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:27.081426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:27.081381    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:28.426657 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:28.426621    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:28.542437 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:28.542389    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:28.665716 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:28.665688    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:28.802787 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:28.802710    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:28.935944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:28.935914    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:29.745303 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:29.745270    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:29.894042 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:29.893985    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:30.343440 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:30.343402    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:30.457666 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:30.457630    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:30.623454 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:30.623422    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:30.762707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:30.762676    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:30.904114 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:30.904007    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:31.121451 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:31.121411    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:31.622860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:31.622820    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:31.809149 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:31.809111    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:31.937817 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:31.937742    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:31.986280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:31.986250    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"]
Nov 02 12:35:31.989903 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:31.989885    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"
Nov 02 12:35:32.017874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.017837    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"]
Nov 02 12:35:32.024738 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.024713    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-628cf9bd\""
Nov 02 12:35:32.024888 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.024790    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-628cf9bd-dockercfg-xmlrt\""
Nov 02 12:35:32.026365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.026342    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"]
Nov 02 12:35:32.030314 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.030296    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"
Nov 02 12:35:32.044755 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.044725    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"]
Nov 02 12:35:32.061873 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.061841    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tbmt\" (UniqueName: \"kubernetes.io/projected/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-kube-api-access-7tbmt\") pod \"payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv\" (UID: \"a44454e4-bc88-4ed6-ba7c-2c7303e745e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"
Nov 02 12:35:32.062110 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.062087    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv\" (UID: \"a44454e4-bc88-4ed6-ba7c-2c7303e745e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"
Nov 02 12:35:32.063816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.063780    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:32.163371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.163340    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv\" (UID: \"a44454e4-bc88-4ed6-ba7c-2c7303e745e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"
Nov 02 12:35:32.163371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.163381    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tbmt\" (UniqueName: \"kubernetes.io/projected/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-kube-api-access-7tbmt\") pod \"payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv\" (UID: \"a44454e4-bc88-4ed6-ba7c-2c7303e745e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"
Nov 02 12:35:32.163578 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.163434    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5skl\" (UniqueName: \"kubernetes.io/projected/143c6461-9fe1-4383-aa8e-5ff3f7760f91-kube-api-access-w5skl\") pod \"payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb\" (UID: \"143c6461-9fe1-4383-aa8e-5ff3f7760f91\") " pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"
Nov 02 12:35:32.163578 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.163457    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c6461-9fe1-4383-aa8e-5ff3f7760f91-plugins-config-volume\") pod \"payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb\" (UID: \"143c6461-9fe1-4383-aa8e-5ff3f7760f91\") " pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"
Nov 02 12:35:32.164105 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.164088    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv\" (UID: \"a44454e4-bc88-4ed6-ba7c-2c7303e745e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"
Nov 02 12:35:32.190900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.190841    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tbmt\" (UniqueName: \"kubernetes.io/projected/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-kube-api-access-7tbmt\") pod \"payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv\" (UID: \"a44454e4-bc88-4ed6-ba7c-2c7303e745e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"
Nov 02 12:35:32.264559 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.264527    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5skl\" (UniqueName: \"kubernetes.io/projected/143c6461-9fe1-4383-aa8e-5ff3f7760f91-kube-api-access-w5skl\") pod \"payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb\" (UID: \"143c6461-9fe1-4383-aa8e-5ff3f7760f91\") " pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"
Nov 02 12:35:32.264719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.264570    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c6461-9fe1-4383-aa8e-5ff3f7760f91-plugins-config-volume\") pod \"payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb\" (UID: \"143c6461-9fe1-4383-aa8e-5ff3f7760f91\") " pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"
Nov 02 12:35:32.265464 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.265444    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c6461-9fe1-4383-aa8e-5ff3f7760f91-plugins-config-volume\") pod \"payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb\" (UID: \"143c6461-9fe1-4383-aa8e-5ff3f7760f91\") " pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"
Nov 02 12:35:32.274277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.274254    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:32.289809 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.289780    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5skl\" (UniqueName: \"kubernetes.io/projected/143c6461-9fe1-4383-aa8e-5ff3f7760f91-kube-api-access-w5skl\") pod \"payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb\" (UID: \"143c6461-9fe1-4383-aa8e-5ff3f7760f91\") " pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"
Nov 02 12:35:32.304642 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.304617    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"
Nov 02 12:35:32.349260 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.349225    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"
Nov 02 12:35:32.603861 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.603831    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"]
Nov 02 12:35:32.604616 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:35:32.604589    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44454e4_bc88_4ed6_ba7c_2c7303e745e8.slice/crio-fc082ed58b9f11ce4ede9d9a2ff34050f73dfaaaa42333b1df25130052baf3f1 WatchSource:0}: Error finding container fc082ed58b9f11ce4ede9d9a2ff34050f73dfaaaa42333b1df25130052baf3f1: Status 404 returned error can't find the container with id fc082ed58b9f11ce4ede9d9a2ff34050f73dfaaaa42333b1df25130052baf3f1
Nov 02 12:35:32.667035 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:32.666981    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"]
Nov 02 12:35:32.667401 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:35:32.667368    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143c6461_9fe1_4383_aa8e_5ff3f7760f91.slice/crio-90cebdc77f304ef056c8b20eab3b71e3f3125879fcb6ccc1da143fc234c04b39 WatchSource:0}: Error finding container 90cebdc77f304ef056c8b20eab3b71e3f3125879fcb6ccc1da143fc234c04b39: Status 404 returned error can't find the container with id 90cebdc77f304ef056c8b20eab3b71e3f3125879fcb6ccc1da143fc234c04b39
Nov 02 12:35:33.398544 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:33.398511    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv" event={"ID":"a44454e4-bc88-4ed6-ba7c-2c7303e745e8","Type":"ContainerStarted","Data":"fc082ed58b9f11ce4ede9d9a2ff34050f73dfaaaa42333b1df25130052baf3f1"}
Nov 02 12:35:33.400884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:33.400856    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb" event={"ID":"143c6461-9fe1-4383-aa8e-5ff3f7760f91","Type":"ContainerStarted","Data":"90cebdc77f304ef056c8b20eab3b71e3f3125879fcb6ccc1da143fc234c04b39"}
Nov 02 12:35:33.476437 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:33.476392    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:34.090052 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:34.090018    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:34.373219 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:34.373175    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:34.566411 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:34.566372    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:35.415934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:35.415866    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb" event={"ID":"143c6461-9fe1-4383-aa8e-5ff3f7760f91","Type":"ContainerStarted","Data":"3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c"}
Nov 02 12:35:35.416139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:35.416026    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"
Nov 02 12:35:35.417520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:35.417499    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv" event={"ID":"a44454e4-bc88-4ed6-ba7c-2c7303e745e8","Type":"ContainerStarted","Data":"5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced"}
Nov 02 12:35:35.417660 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:35.417642    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"
Nov 02 12:35:35.460623 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:35.460569    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb" podStartSLOduration=2.306731703 podStartE2EDuration="4.46055485s" totalImagesPullingTime="2.153823147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:35.459370089 +0000 UTC m=+481.566072941" watchObservedRunningTime="2026-11-02 12:35:35.46055485 +0000 UTC m=+481.567257695"
Nov 02 12:35:35.503203 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:35.503141    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv" podStartSLOduration=2.368603561 podStartE2EDuration="4.503112261s" totalImagesPullingTime="2.1345087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:35.502419546 +0000 UTC m=+481.609122398" watchObservedRunningTime="2026-11-02 12:35:35.503112261 +0000 UTC m=+481.609815109"
Nov 02 12:35:35.923919 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:35.923888    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:36.055495 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.055464    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:36.183062 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.182963    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:36.300568 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.300540    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"]
Nov 02 12:35:36.304511 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.304494    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"
Nov 02 12:35:36.319369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.319345    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:36.320754 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.320729    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlvkl\" (UniqueName: \"kubernetes.io/projected/45b4a94a-8cdb-4825-a30f-f6594a3f5940-kube-api-access-mlvkl\") pod \"payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh\" (UID: \"45b4a94a-8cdb-4825-a30f-f6594a3f5940\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"
Nov 02 12:35:36.320859 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.320843    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b4a94a-8cdb-4825-a30f-f6594a3f5940-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh\" (UID: \"45b4a94a-8cdb-4825-a30f-f6594a3f5940\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"
Nov 02 12:35:36.324208 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.324180    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"]
Nov 02 12:35:36.325169 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.325147    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-2f0bef09-dockercfg-6572n\""
Nov 02 12:35:36.325377 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.325339    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2f0bef09\""
Nov 02 12:35:36.376350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.376327    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"]
Nov 02 12:35:36.379884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.379868    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"
Nov 02 12:35:36.402229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.401405    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"]
Nov 02 12:35:36.422184 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.422150    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0c2ad2-30ca-489c-b92e-2e07babd8a73-plugins-config-volume\") pod \"payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf\" (UID: \"de0c2ad2-30ca-489c-b92e-2e07babd8a73\") " pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"
Nov 02 12:35:36.422334 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.422203    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lnxb\" (UniqueName: \"kubernetes.io/projected/de0c2ad2-30ca-489c-b92e-2e07babd8a73-kube-api-access-6lnxb\") pod \"payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf\" (UID: \"de0c2ad2-30ca-489c-b92e-2e07babd8a73\") " pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"
Nov 02 12:35:36.422334 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.422296    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvkl\" (UniqueName: \"kubernetes.io/projected/45b4a94a-8cdb-4825-a30f-f6594a3f5940-kube-api-access-mlvkl\") pod \"payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh\" (UID: \"45b4a94a-8cdb-4825-a30f-f6594a3f5940\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"
Nov 02 12:35:36.422416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.422397    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b4a94a-8cdb-4825-a30f-f6594a3f5940-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh\" (UID: \"45b4a94a-8cdb-4825-a30f-f6594a3f5940\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"
Nov 02 12:35:36.423359 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.423335    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b4a94a-8cdb-4825-a30f-f6594a3f5940-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh\" (UID: \"45b4a94a-8cdb-4825-a30f-f6594a3f5940\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"
Nov 02 12:35:36.447297 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.447234    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlvkl\" (UniqueName: \"kubernetes.io/projected/45b4a94a-8cdb-4825-a30f-f6594a3f5940-kube-api-access-mlvkl\") pod \"payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh\" (UID: \"45b4a94a-8cdb-4825-a30f-f6594a3f5940\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"
Nov 02 12:35:36.522911 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.522879    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lnxb\" (UniqueName: \"kubernetes.io/projected/de0c2ad2-30ca-489c-b92e-2e07babd8a73-kube-api-access-6lnxb\") pod \"payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf\" (UID: \"de0c2ad2-30ca-489c-b92e-2e07babd8a73\") " pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"
Nov 02 12:35:36.523163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.523139    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0c2ad2-30ca-489c-b92e-2e07babd8a73-plugins-config-volume\") pod \"payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf\" (UID: \"de0c2ad2-30ca-489c-b92e-2e07babd8a73\") " pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"
Nov 02 12:35:36.523703 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.523685    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0c2ad2-30ca-489c-b92e-2e07babd8a73-plugins-config-volume\") pod \"payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf\" (UID: \"de0c2ad2-30ca-489c-b92e-2e07babd8a73\") " pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"
Nov 02 12:35:36.549934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.549911    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lnxb\" (UniqueName: \"kubernetes.io/projected/de0c2ad2-30ca-489c-b92e-2e07babd8a73-kube-api-access-6lnxb\") pod \"payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf\" (UID: \"de0c2ad2-30ca-489c-b92e-2e07babd8a73\") " pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"
Nov 02 12:35:36.617607 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.617573    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"
Nov 02 12:35:36.696619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.696408    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"
Nov 02 12:35:36.928290 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:36.928256    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"]
Nov 02 12:35:36.928734 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:35:36.928317    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b4a94a_8cdb_4825_a30f_f6594a3f5940.slice/crio-c7c4274722d1acbcb169b6802c2ba0b4f27a3bb9437fda265e99161e25c899ae WatchSource:0}: Error finding container c7c4274722d1acbcb169b6802c2ba0b4f27a3bb9437fda265e99161e25c899ae: Status 404 returned error can't find the container with id c7c4274722d1acbcb169b6802c2ba0b4f27a3bb9437fda265e99161e25c899ae
Nov 02 12:35:37.001414 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:37.001390    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"]
Nov 02 12:35:37.002566 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:35:37.002532    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0c2ad2_30ca_489c_b92e_2e07babd8a73.slice/crio-2af6d2ac6c731bba455cf65703a39926ab5bcfec5289c8aab48a24c5035babbb WatchSource:0}: Error finding container 2af6d2ac6c731bba455cf65703a39926ab5bcfec5289c8aab48a24c5035babbb: Status 404 returned error can't find the container with id 2af6d2ac6c731bba455cf65703a39926ab5bcfec5289c8aab48a24c5035babbb
Nov 02 12:35:37.431722 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:37.431684    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh" event={"ID":"45b4a94a-8cdb-4825-a30f-f6594a3f5940","Type":"ContainerStarted","Data":"1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d"}
Nov 02 12:35:37.431934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:37.431777    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh" event={"ID":"45b4a94a-8cdb-4825-a30f-f6594a3f5940","Type":"ContainerStarted","Data":"c7c4274722d1acbcb169b6802c2ba0b4f27a3bb9437fda265e99161e25c899ae"}
Nov 02 12:35:37.431934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:37.431810    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"
Nov 02 12:35:37.433509 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:37.433475    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf" event={"ID":"de0c2ad2-30ca-489c-b92e-2e07babd8a73","Type":"ContainerStarted","Data":"7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec"}
Nov 02 12:35:37.433639 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:37.433539    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"
Nov 02 12:35:37.433639 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:37.433564    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf" event={"ID":"de0c2ad2-30ca-489c-b92e-2e07babd8a73","Type":"ContainerStarted","Data":"2af6d2ac6c731bba455cf65703a39926ab5bcfec5289c8aab48a24c5035babbb"}
Nov 02 12:35:37.476495 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:37.476443    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh" podStartSLOduration=1.476430442 podStartE2EDuration="1.476430442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:37.474893077 +0000 UTC m=+483.581595929" watchObservedRunningTime="2026-11-02 12:35:37.476430442 +0000 UTC m=+483.583133292"
Nov 02 12:35:37.537634 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:37.537584    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf" podStartSLOduration=1.537569703 podStartE2EDuration="1.537569703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:37.535374314 +0000 UTC m=+483.642077167" watchObservedRunningTime="2026-11-02 12:35:37.537569703 +0000 UTC m=+483.644272554"
Nov 02 12:35:37.882832 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:37.882795    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:39.541936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:39.541901    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:41.262659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:41.262628    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:43.127895 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:43.127856    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:43.257378 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:43.257337    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:43.376917 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:43.376865    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:44.779711 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:44.779677    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:44.822462 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:35:44.822428    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:35:45.991611 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:45.991582    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"]
Nov 02 12:35:45.995699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:45.995676    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:46.012845 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.012810    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:35:46.026605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.026580    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:35:46.028282 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.028256    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"]
Nov 02 12:35:46.107856 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.107822    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhqkp\" (UniqueName: \"kubernetes.io/projected/9e75dc59-78a0-442f-87ee-22888bfc9cd8-kube-api-access-mhqkp\") pod \"maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47\" (UID: \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:46.108013 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.107914    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47\" (UID: \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:46.108013 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.107948    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47\" (UID: \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:46.209656 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.209615    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47\" (UID: \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:46.209803 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.209727    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhqkp\" (UniqueName: \"kubernetes.io/projected/9e75dc59-78a0-442f-87ee-22888bfc9cd8-kube-api-access-mhqkp\") pod \"maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47\" (UID: \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:46.209879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.209805    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47\" (UID: \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:46.210367 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.210345    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47\" (UID: \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:46.212296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.212269    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47\" (UID: \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:46.234465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.234429    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhqkp\" (UniqueName: \"kubernetes.io/projected/9e75dc59-78a0-442f-87ee-22888bfc9cd8-kube-api-access-mhqkp\") pod \"maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47\" (UID: \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:46.308709 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.308633    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:46.426937 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.426907    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"
Nov 02 12:35:46.427115 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.426981    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"
Nov 02 12:35:46.468209 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.467421    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:46.622477 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:46.622453    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"]
Nov 02 12:35:46.623524 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:35:46.623495    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e75dc59_78a0_442f_87ee_22888bfc9cd8.slice/crio-82582c3bbc7d3fdc89f5ebd4b8a629a18fb8d1f6f876ac3189ed7bb61e69e1a1 WatchSource:0}: Error finding container 82582c3bbc7d3fdc89f5ebd4b8a629a18fb8d1f6f876ac3189ed7bb61e69e1a1: Status 404 returned error can't find the container with id 82582c3bbc7d3fdc89f5ebd4b8a629a18fb8d1f6f876ac3189ed7bb61e69e1a1
Nov 02 12:35:47.489568 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:47.489527    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47" event={"ID":"9e75dc59-78a0-442f-87ee-22888bfc9cd8","Type":"ContainerStarted","Data":"82582c3bbc7d3fdc89f5ebd4b8a629a18fb8d1f6f876ac3189ed7bb61e69e1a1"}
Nov 02 12:35:48.442267 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:48.442239    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"
Nov 02 12:35:48.442453 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:48.442310    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"
Nov 02 12:35:48.665973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:48.665930    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:49.502981 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:49.502947    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47" event={"ID":"9e75dc59-78a0-442f-87ee-22888bfc9cd8","Type":"ContainerStarted","Data":"db1536a189ef6ff6de542ba8ba52178fdc9f72052b919b31c7d4990121ebc230"}
Nov 02 12:35:49.547529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:49.547464    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47" podStartSLOduration=2.455152674 podStartE2EDuration="4.547447306s" totalImagesPullingTime="2.092294632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:35:49.545692323 +0000 UTC m=+495.652395173" watchObservedRunningTime="2026-11-02 12:35:49.547447306 +0000 UTC m=+495.654150157"
Nov 02 12:35:50.412884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:50.412848    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:50.509623 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:50.509584    5879 generic.go:356] "Generic (PLEG): container finished" podID="9e75dc59-78a0-442f-87ee-22888bfc9cd8" containerID="db1536a189ef6ff6de542ba8ba52178fdc9f72052b919b31c7d4990121ebc230" exitCode=0
Nov 02 12:35:50.509788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:50.509674    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47" event={"ID":"9e75dc59-78a0-442f-87ee-22888bfc9cd8","Type":"ContainerDied","Data":"db1536a189ef6ff6de542ba8ba52178fdc9f72052b919b31c7d4990121ebc230"}
Nov 02 12:35:51.426832 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:51.426787    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:51.554215 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:51.554185    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:51.662089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:51.662065    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:51.770622 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:51.770540    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e75dc59-78a0-442f-87ee-22888bfc9cd8-kube-api-access-mhqkp\" (UniqueName: \"kubernetes.io/projected/9e75dc59-78a0-442f-87ee-22888bfc9cd8-kube-api-access-mhqkp\") pod \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\" (UID: \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\") "
Nov 02 12:35:51.770774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:51.770689    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-xks-ca\") pod \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\" (UID: \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\") "
Nov 02 12:35:51.770774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:51.770721    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-openshift-ca\") pod \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\" (UID: \"9e75dc59-78a0-442f-87ee-22888bfc9cd8\") "
Nov 02 12:35:51.771132 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:51.771102    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-openshift-ca" pod "9e75dc59-78a0-442f-87ee-22888bfc9cd8" (UID: "9e75dc59-78a0-442f-87ee-22888bfc9cd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:35:51.771266 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:51.771198    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:35:51.772937 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:51.772919    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-xks-ca" pod "9e75dc59-78a0-442f-87ee-22888bfc9cd8" (UID: "9e75dc59-78a0-442f-87ee-22888bfc9cd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:35:51.773123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:51.773101    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e75dc59-78a0-442f-87ee-22888bfc9cd8-kube-api-access-mhqkp" pod "9e75dc59-78a0-442f-87ee-22888bfc9cd8" (UID: "9e75dc59-78a0-442f-87ee-22888bfc9cd8"). InnerVolumeSpecName "kube-api-access-mhqkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:35:51.871948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:51.871916    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e75dc59-78a0-442f-87ee-22888bfc9cd8-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:35:51.871948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:51.871941    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhqkp\" (UniqueName: \"kubernetes.io/projected/9e75dc59-78a0-442f-87ee-22888bfc9cd8-kube-api-access-mhqkp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:35:52.034443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:52.034411    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:52.523836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:52.523803    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"
Nov 02 12:35:52.524352 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:52.523831    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47" event={"ID":"9e75dc59-78a0-442f-87ee-22888bfc9cd8","Type":"ContainerDied","Data":"82582c3bbc7d3fdc89f5ebd4b8a629a18fb8d1f6f876ac3189ed7bb61e69e1a1"}
Nov 02 12:35:52.524352 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:52.523922    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82582c3bbc7d3fdc89f5ebd4b8a629a18fb8d1f6f876ac3189ed7bb61e69e1a1"
Nov 02 12:35:52.931177 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:52.931140    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:53.056541 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:53.056492    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:53.801148 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:53.801097    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:55.165874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:55.165837    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:55.683726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:55.683687    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:55.806221 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:55.806166    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:56.034400 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:56.034308    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:56.854439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:56.854403    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:57.010979 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:57.010942    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:57.139783 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:57.139748    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:57.300364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:57.300328    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:57.422058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:57.421949    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:57.542364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:57.542329    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:57.775380 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:57.775289    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:57.910237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:57.910202    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:58.029886 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:58.029797    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:58.278358 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:58.278322    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:58.399697 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:58.399659    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:58.422397 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:58.422366    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"]
Nov 02 12:35:58.436013 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:58.435965    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f080cb5f-rmj47"]
Nov 02 12:35:58.569968 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:58.569938    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:58.692327 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:58.692245    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:59.486694 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:59.486654    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:35:59.726772 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:35:59.726736    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:00.000352 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:00.000319    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:00.017705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:00.017665    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e75dc59-78a0-442f-87ee-22888bfc9cd8" path="/var/lib/kubelet/pods/9e75dc59-78a0-442f-87ee-22888bfc9cd8/volumes"
Nov 02 12:36:00.222679 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:00.222646    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:00.560293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:00.560234    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:00.677499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:00.677453    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:01.077126 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:01.077082    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:01.317578 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:01.317542    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:01.715169 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:01.715129    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:01.851239 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:01.851197    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:02.422151 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:02.422124    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:03.408383 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:03.408346    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:03.554460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:03.554422    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:03.878336 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:03.878307    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:04.015410 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:04.015364    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:04.141472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:04.141382    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:04.582637 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:04.582558    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:04.714954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:04.714923    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:04.786644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:04.786610    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"]
Nov 02 12:36:04.828218 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:04.828183    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"]
Nov 02 12:36:04.959416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:04.959378    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:05.083785 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:05.083747    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:05.930556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:05.930524    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:07.636451 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:07.636416    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:09.127426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:09.127373    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:09.637329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:09.637297    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:09.772860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:09.772815    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:09.787673 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:09.787641    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb" podUID="143c6461-9fe1-4383-aa8e-5ff3f7760f91" containerName="payload-processing" containerID="cri-o://3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c" gracePeriod=25
Nov 02 12:36:09.828519 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:09.828482    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv" podUID="a44454e4-bc88-4ed6-ba7c-2c7303e745e8" containerName="payload-pre-processing" containerID="cri-o://5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced" gracePeriod=25
Nov 02 12:36:09.893727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:09.893668    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:10.022333 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.022248    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:10.081658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.081627    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"
Nov 02 12:36:10.117939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.117919    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"
Nov 02 12:36:10.265341 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.265258    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-plugins-config-volume\") pod \"a44454e4-bc88-4ed6-ba7c-2c7303e745e8\" (UID: \"a44454e4-bc88-4ed6-ba7c-2c7303e745e8\") "
Nov 02 12:36:10.265341 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.265329    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143c6461-9fe1-4383-aa8e-5ff3f7760f91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c6461-9fe1-4383-aa8e-5ff3f7760f91-plugins-config-volume\") pod \"143c6461-9fe1-4383-aa8e-5ff3f7760f91\" (UID: \"143c6461-9fe1-4383-aa8e-5ff3f7760f91\") "
Nov 02 12:36:10.265857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.265357    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143c6461-9fe1-4383-aa8e-5ff3f7760f91-kube-api-access-w5skl\" (UniqueName: \"kubernetes.io/projected/143c6461-9fe1-4383-aa8e-5ff3f7760f91-kube-api-access-w5skl\") pod \"143c6461-9fe1-4383-aa8e-5ff3f7760f91\" (UID: \"143c6461-9fe1-4383-aa8e-5ff3f7760f91\") "
Nov 02 12:36:10.265857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.265416    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-kube-api-access-7tbmt\" (UniqueName: \"kubernetes.io/projected/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-kube-api-access-7tbmt\") pod \"a44454e4-bc88-4ed6-ba7c-2c7303e745e8\" (UID: \"a44454e4-bc88-4ed6-ba7c-2c7303e745e8\") "
Nov 02 12:36:10.265857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.265681    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-plugins-config-volume" pod "a44454e4-bc88-4ed6-ba7c-2c7303e745e8" (UID: "a44454e4-bc88-4ed6-ba7c-2c7303e745e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:10.265857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.265760    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143c6461-9fe1-4383-aa8e-5ff3f7760f91-plugins-config-volume" pod "143c6461-9fe1-4383-aa8e-5ff3f7760f91" (UID: "143c6461-9fe1-4383-aa8e-5ff3f7760f91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:10.267778 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.267760    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-kube-api-access-7tbmt" pod "a44454e4-bc88-4ed6-ba7c-2c7303e745e8" (UID: "a44454e4-bc88-4ed6-ba7c-2c7303e745e8"). InnerVolumeSpecName "kube-api-access-7tbmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:10.267842 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.267787    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143c6461-9fe1-4383-aa8e-5ff3f7760f91-kube-api-access-w5skl" pod "143c6461-9fe1-4383-aa8e-5ff3f7760f91" (UID: "143c6461-9fe1-4383-aa8e-5ff3f7760f91"). InnerVolumeSpecName "kube-api-access-w5skl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:10.369182 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.367035    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5skl\" (UniqueName: \"kubernetes.io/projected/143c6461-9fe1-4383-aa8e-5ff3f7760f91-kube-api-access-w5skl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:36:10.369182 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.367065    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tbmt\" (UniqueName: \"kubernetes.io/projected/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-kube-api-access-7tbmt\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:36:10.369182 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.367078    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44454e4-bc88-4ed6-ba7c-2c7303e745e8-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:36:10.369182 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.367090    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/143c6461-9fe1-4383-aa8e-5ff3f7760f91-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:36:10.451083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.451047    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:10.619448 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.619416    5879 generic.go:356] "Generic (PLEG): container finished" podID="a44454e4-bc88-4ed6-ba7c-2c7303e745e8" containerID="5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced" exitCode=0
Nov 02 12:36:10.619620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.619470    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"
Nov 02 12:36:10.619620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.619476    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv" event={"ID":"a44454e4-bc88-4ed6-ba7c-2c7303e745e8","Type":"ContainerDied","Data":"5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced"}
Nov 02 12:36:10.619620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.619527    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv" event={"ID":"a44454e4-bc88-4ed6-ba7c-2c7303e745e8","Type":"ContainerDied","Data":"fc082ed58b9f11ce4ede9d9a2ff34050f73dfaaaa42333b1df25130052baf3f1"}
Nov 02 12:36:10.619620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.619548    5879 scope.go:122] "RemoveContainer" containerID="5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced"
Nov 02 12:36:10.621003 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.620975    5879 generic.go:356] "Generic (PLEG): container finished" podID="143c6461-9fe1-4383-aa8e-5ff3f7760f91" containerID="3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c" exitCode=0
Nov 02 12:36:10.621151 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.621058    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"
Nov 02 12:36:10.621151 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.621067    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb" event={"ID":"143c6461-9fe1-4383-aa8e-5ff3f7760f91","Type":"ContainerDied","Data":"3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c"}
Nov 02 12:36:10.621282 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.621151    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb" event={"ID":"143c6461-9fe1-4383-aa8e-5ff3f7760f91","Type":"ContainerDied","Data":"90cebdc77f304ef056c8b20eab3b71e3f3125879fcb6ccc1da143fc234c04b39"}
Nov 02 12:36:10.628700 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.628682    5879 scope.go:122] "RemoveContainer" containerID="5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced"
Nov 02 12:36:10.629058 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:36:10.629036    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced\": container with ID starting with 5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced not found: ID does not exist" containerID="5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced"
Nov 02 12:36:10.629117 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.629068    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced"} err="failed to get container status \"5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced\": rpc error: code = NotFound desc = could not find container \"5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced\": container with ID starting with 5dfdf0d49ed20fa8bcd6ebc02bc4717ceceead5f492067d0ef257fa4fd426ced not found: ID does not exist"
Nov 02 12:36:10.629117 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.629086    5879 scope.go:122] "RemoveContainer" containerID="3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c"
Nov 02 12:36:10.637193 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.637179    5879 scope.go:122] "RemoveContainer" containerID="3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c"
Nov 02 12:36:10.637433 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:36:10.637412    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c\": container with ID starting with 3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c not found: ID does not exist" containerID="3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c"
Nov 02 12:36:10.637492 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.637445    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c"} err="failed to get container status \"3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c\": rpc error: code = NotFound desc = could not find container \"3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c\": container with ID starting with 3d54020d1df3dc3de5eb20e985634a0d82d7443719f3f345b0465423aad91c3c not found: ID does not exist"
Nov 02 12:36:10.674788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.674764    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"]
Nov 02 12:36:10.688412 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.688388    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-628cf9bd-54c8dc87f9-khxwv"]
Nov 02 12:36:10.725232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.725197    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"]
Nov 02 12:36:10.729960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.729940    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-628cf9bd-57b7789fcf-djlqb"]
Nov 02 12:36:10.912899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.912826    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"]
Nov 02 12:36:10.942745 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:10.942716    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"]
Nov 02 12:36:11.074352 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:11.074324    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:11.198692 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:11.198613    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:11.343517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:11.343485    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:11.664464 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:11.664423    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:11.797479 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:11.797448    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:11.926774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:11.926689    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:12.015468 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:12.015435    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143c6461-9fe1-4383-aa8e-5ff3f7760f91" path="/var/lib/kubelet/pods/143c6461-9fe1-4383-aa8e-5ff3f7760f91/volumes"
Nov 02 12:36:12.015839 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:12.015825    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44454e4-bc88-4ed6-ba7c-2c7303e745e8" path="/var/lib/kubelet/pods/a44454e4-bc88-4ed6-ba7c-2c7303e745e8/volumes"
Nov 02 12:36:13.456317 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:13.456279    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:14.824668 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:36:14.824632    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:36:15.180667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:15.180633    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:15.667685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:15.667651    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:15.791371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:15.791328    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:15.914017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:15.913953    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf" podUID="de0c2ad2-30ca-489c-b92e-2e07babd8a73" containerName="payload-processing" containerID="cri-o://7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec" gracePeriod=25
Nov 02 12:36:15.943683 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:15.943634    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh" podUID="45b4a94a-8cdb-4825-a30f-f6594a3f5940" containerName="payload-pre-processing" containerID="cri-o://1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d" gracePeriod=25
Nov 02 12:36:16.191901 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.191879    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"
Nov 02 12:36:16.215550 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.215531    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"
Nov 02 12:36:16.216985 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.216963    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0c2ad2-30ca-489c-b92e-2e07babd8a73-kube-api-access-6lnxb\" (UniqueName: \"kubernetes.io/projected/de0c2ad2-30ca-489c-b92e-2e07babd8a73-kube-api-access-6lnxb\") pod \"de0c2ad2-30ca-489c-b92e-2e07babd8a73\" (UID: \"de0c2ad2-30ca-489c-b92e-2e07babd8a73\") "
Nov 02 12:36:16.217156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.217142    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de0c2ad2-30ca-489c-b92e-2e07babd8a73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0c2ad2-30ca-489c-b92e-2e07babd8a73-plugins-config-volume\") pod \"de0c2ad2-30ca-489c-b92e-2e07babd8a73\" (UID: \"de0c2ad2-30ca-489c-b92e-2e07babd8a73\") "
Nov 02 12:36:16.217493 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.217472    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0c2ad2-30ca-489c-b92e-2e07babd8a73-plugins-config-volume" pod "de0c2ad2-30ca-489c-b92e-2e07babd8a73" (UID: "de0c2ad2-30ca-489c-b92e-2e07babd8a73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:16.218982 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.218950    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0c2ad2-30ca-489c-b92e-2e07babd8a73-kube-api-access-6lnxb" pod "de0c2ad2-30ca-489c-b92e-2e07babd8a73" (UID: "de0c2ad2-30ca-489c-b92e-2e07babd8a73"). InnerVolumeSpecName "kube-api-access-6lnxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:16.318738 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.318708    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45b4a94a-8cdb-4825-a30f-f6594a3f5940-kube-api-access-mlvkl\" (UniqueName: \"kubernetes.io/projected/45b4a94a-8cdb-4825-a30f-f6594a3f5940-kube-api-access-mlvkl\") pod \"45b4a94a-8cdb-4825-a30f-f6594a3f5940\" (UID: \"45b4a94a-8cdb-4825-a30f-f6594a3f5940\") "
Nov 02 12:36:16.318900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.318787    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45b4a94a-8cdb-4825-a30f-f6594a3f5940-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b4a94a-8cdb-4825-a30f-f6594a3f5940-plugins-config-volume\") pod \"45b4a94a-8cdb-4825-a30f-f6594a3f5940\" (UID: \"45b4a94a-8cdb-4825-a30f-f6594a3f5940\") "
Nov 02 12:36:16.319062 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.319043    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lnxb\" (UniqueName: \"kubernetes.io/projected/de0c2ad2-30ca-489c-b92e-2e07babd8a73-kube-api-access-6lnxb\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:36:16.319107 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.319070    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0c2ad2-30ca-489c-b92e-2e07babd8a73-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:36:16.319190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.319175    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b4a94a-8cdb-4825-a30f-f6594a3f5940-plugins-config-volume" pod "45b4a94a-8cdb-4825-a30f-f6594a3f5940" (UID: "45b4a94a-8cdb-4825-a30f-f6594a3f5940"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:36:16.320851 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.320831    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b4a94a-8cdb-4825-a30f-f6594a3f5940-kube-api-access-mlvkl" pod "45b4a94a-8cdb-4825-a30f-f6594a3f5940" (UID: "45b4a94a-8cdb-4825-a30f-f6594a3f5940"). InnerVolumeSpecName "kube-api-access-mlvkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:36:16.419522 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.419448    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlvkl\" (UniqueName: \"kubernetes.io/projected/45b4a94a-8cdb-4825-a30f-f6594a3f5940-kube-api-access-mlvkl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:36:16.419522 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.419471    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45b4a94a-8cdb-4825-a30f-f6594a3f5940-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:36:16.587914 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.587884    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:16.655464 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.655431    5879 generic.go:356] "Generic (PLEG): container finished" podID="45b4a94a-8cdb-4825-a30f-f6594a3f5940" containerID="1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d" exitCode=0
Nov 02 12:36:16.655617 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.655477    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"
Nov 02 12:36:16.655617 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.655521    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh" event={"ID":"45b4a94a-8cdb-4825-a30f-f6594a3f5940","Type":"ContainerDied","Data":"1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d"}
Nov 02 12:36:16.655617 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.655577    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh" event={"ID":"45b4a94a-8cdb-4825-a30f-f6594a3f5940","Type":"ContainerDied","Data":"c7c4274722d1acbcb169b6802c2ba0b4f27a3bb9437fda265e99161e25c899ae"}
Nov 02 12:36:16.655617 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.655614    5879 scope.go:122] "RemoveContainer" containerID="1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d"
Nov 02 12:36:16.656840 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.656817    5879 generic.go:356] "Generic (PLEG): container finished" podID="de0c2ad2-30ca-489c-b92e-2e07babd8a73" containerID="7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec" exitCode=0
Nov 02 12:36:16.656937 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.656853    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf" event={"ID":"de0c2ad2-30ca-489c-b92e-2e07babd8a73","Type":"ContainerDied","Data":"7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec"}
Nov 02 12:36:16.656937 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.656871    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"
Nov 02 12:36:16.656937 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.656889    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf" event={"ID":"de0c2ad2-30ca-489c-b92e-2e07babd8a73","Type":"ContainerDied","Data":"2af6d2ac6c731bba455cf65703a39926ab5bcfec5289c8aab48a24c5035babbb"}
Nov 02 12:36:16.665272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.665248    5879 scope.go:122] "RemoveContainer" containerID="1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d"
Nov 02 12:36:16.665665 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:36:16.665623    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d\": container with ID starting with 1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d not found: ID does not exist" containerID="1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d"
Nov 02 12:36:16.665753 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.665673    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d"} err="failed to get container status \"1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d\": rpc error: code = NotFound desc = could not find container \"1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d\": container with ID starting with 1217d13fbd9a7bdaaa7e24fe05032346b84f7071d915e6f5468a9b63f07e756d not found: ID does not exist"
Nov 02 12:36:16.665753 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.665697    5879 scope.go:122] "RemoveContainer" containerID="7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec"
Nov 02 12:36:16.673908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.673892    5879 scope.go:122] "RemoveContainer" containerID="7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec"
Nov 02 12:36:16.674190 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:36:16.674166    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec\": container with ID starting with 7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec not found: ID does not exist" containerID="7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec"
Nov 02 12:36:16.674269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.674208    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec"} err="failed to get container status \"7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec\": rpc error: code = NotFound desc = could not find container \"7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec\": container with ID starting with 7e901c6b00ba6dffccd4b2e32255c16807dc27a7b016758bb3caf494b8c71fec not found: ID does not exist"
Nov 02 12:36:16.710754 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.710735    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"]
Nov 02 12:36:16.725460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.725436    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-2f0bef09-559657ddff-w8cdf"]
Nov 02 12:36:16.750502 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.750476    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"]
Nov 02 12:36:16.755293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:16.755269    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f0bef09-6d796cc9ff-ljpnh"]
Nov 02 12:36:17.239507 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:17.239423    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:17.416170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:17.416143    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:17.565139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:17.565045    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:17.693553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:17.693522    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:18.015980 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:18.015945    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45b4a94a-8cdb-4825-a30f-f6594a3f5940" path="/var/lib/kubelet/pods/45b4a94a-8cdb-4825-a30f-f6594a3f5940/volumes"
Nov 02 12:36:18.016356 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:18.016342    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de0c2ad2-30ca-489c-b92e-2e07babd8a73" path="/var/lib/kubelet/pods/de0c2ad2-30ca-489c-b92e-2e07babd8a73/volumes"
Nov 02 12:36:18.433402 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:18.433361    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:18.933491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:18.933456    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:19.054351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:19.054317    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:19.182511 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:19.182469    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:20.863278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:20.863243    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:21.481774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:21.481699    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:21.611552 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:21.611514    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:22.732111 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:22.732080    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:22.853444 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:22.853407    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:23.081184 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:23.081094    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:23.220047 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:23.219979    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:23.922718 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:23.922679    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:24.450561 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:24.450524    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:24.578585 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:24.578548    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:24.780808 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:24.780724    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:25.121460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:25.121429    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:25.252259 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:25.252224    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:25.395386 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:25.395307    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:25.620638 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:25.620603    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:25.868827 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:25.868794    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:26.353887 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:26.353768    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:26.887462 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:26.887426    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:27.470535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:27.470485    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:27.599120 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:27.599083    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:27.726405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:27.726310    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:28.061259 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:28.061165    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:29.106792 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.106756    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"]
Nov 02 12:36:29.107534 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107511    5879 cpu_manager.go:414] "containerMap: removing container" podUID="de0c2ad2-30ca-489c-b92e-2e07babd8a73" containerName="payload-processing"
Nov 02 12:36:29.107596 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107538    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0c2ad2-30ca-489c-b92e-2e07babd8a73" containerName="payload-processing"
Nov 02 12:36:29.107596 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107562    5879 cpu_manager.go:414] "containerMap: removing container" podUID="45b4a94a-8cdb-4825-a30f-f6594a3f5940" containerName="payload-pre-processing"
Nov 02 12:36:29.107596 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107571    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b4a94a-8cdb-4825-a30f-f6594a3f5940" containerName="payload-pre-processing"
Nov 02 12:36:29.107596 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107591    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9e75dc59-78a0-442f-87ee-22888bfc9cd8" containerName="revoke-keys"
Nov 02 12:36:29.107731 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107599    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e75dc59-78a0-442f-87ee-22888bfc9cd8" containerName="revoke-keys"
Nov 02 12:36:29.107731 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107630    5879 cpu_manager.go:414] "containerMap: removing container" podUID="a44454e4-bc88-4ed6-ba7c-2c7303e745e8" containerName="payload-pre-processing"
Nov 02 12:36:29.107731 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107638    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44454e4-bc88-4ed6-ba7c-2c7303e745e8" containerName="payload-pre-processing"
Nov 02 12:36:29.107731 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107662    5879 cpu_manager.go:414] "containerMap: removing container" podUID="143c6461-9fe1-4383-aa8e-5ff3f7760f91" containerName="payload-processing"
Nov 02 12:36:29.107731 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107670    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143c6461-9fe1-4383-aa8e-5ff3f7760f91" containerName="payload-processing"
Nov 02 12:36:29.107874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107788    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b4a94a-8cdb-4825-a30f-f6594a3f5940" containerName="payload-pre-processing"
Nov 02 12:36:29.107874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107801    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e75dc59-78a0-442f-87ee-22888bfc9cd8" containerName="revoke-keys"
Nov 02 12:36:29.107874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107814    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44454e4-bc88-4ed6-ba7c-2c7303e745e8" containerName="payload-pre-processing"
Nov 02 12:36:29.107874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107825    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="143c6461-9fe1-4383-aa8e-5ff3f7760f91" containerName="payload-processing"
Nov 02 12:36:29.107874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.107839    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0c2ad2-30ca-489c-b92e-2e07babd8a73" containerName="payload-processing"
Nov 02 12:36:29.111498 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.111473    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.142387 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.142354    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-95ee26\"/\"default-dockercfg-gm7qj\""
Nov 02 12:36:29.142387 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.142390    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-95ee26\"/\"kube-root-ca.crt\""
Nov 02 12:36:29.142627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.142414    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-95ee26\"/\"openshift-service-ca.crt\""
Nov 02 12:36:29.142829 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.142803    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-95ee26\"/\"simulator-w2-95ee26-kserve-self-signed-certs\""
Nov 02 12:36:29.145466 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.145444    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"]
Nov 02 12:36:29.246639 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.246596    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:29.253056 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.253024    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-home\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.253196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.253138    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8qzm\" (UniqueName: \"kubernetes.io/projected/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-kube-api-access-r8qzm\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.253260 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.253246    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-model-cache\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.253306 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.253291    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tls-certs\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.253416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.253392    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-dshm\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.253454 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.253435    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tmp-dir\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.355042 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.354977    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-model-cache\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.355237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.355080    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tls-certs\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.355237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.355134    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-dshm\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.355237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.355156    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tmp-dir\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.355237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.355219    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-home\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.355458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.355296    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8qzm\" (UniqueName: \"kubernetes.io/projected/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-kube-api-access-r8qzm\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.355516 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.355462    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-model-cache\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.355578 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.355555    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tmp-dir\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.355661 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.355636    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-home\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.357791 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.357734    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-dshm\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.358399 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.358377    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tls-certs\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.380522 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.380490    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8qzm\" (UniqueName: \"kubernetes.io/projected/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-kube-api-access-r8qzm\") pod \"simulator-w2-95ee26-kserve-54c446c865-tbvjb\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") " pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.431036 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.425626    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:29.734471 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.734434    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"]
Nov 02 12:36:29.759578 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:29.759535    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:30.735171 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:30.735119    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" event={"ID":"c2514e25-9ef1-4ff7-b352-b3eff9cecb76","Type":"ContainerStarted","Data":"9e185482927bddd88497453f1d8e099a49c35316720d1194fa87978b0b67bd79"}
Nov 02 12:36:31.336777 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:31.336739    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:31.462885 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:31.462772    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:31.605070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:31.605033    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:31.742138 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:31.742039    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" event={"ID":"c2514e25-9ef1-4ff7-b352-b3eff9cecb76","Type":"ContainerStarted","Data":"854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73"}
Nov 02 12:36:31.787423 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:31.787368    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podStartSLOduration=1.419673332 podStartE2EDuration="2.787352092s" totalImagesPullingTime="1.36767876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:31.786146264 +0000 UTC m=+537.892849128" watchObservedRunningTime="2026-11-02 12:36:31.787352092 +0000 UTC m=+537.894054943"
Nov 02 12:36:32.426788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:32.426752    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:32.961377 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:32.961340    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:33.602385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:33.602354    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:34.122860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:34.122829    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:34.241902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:34.241867    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:34.631579 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:34.631542    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:34.756679 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:34.756650    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:35.673960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:35.673907    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:36.307806 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:36.307767    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:36.450742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:36.450708    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:36.580348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:36.580257    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:37.030425 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.030394    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"]
Nov 02 12:36:37.034552 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.034523    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.053140 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.053092    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"openshift-service-ca.crt\""
Nov 02 12:36:37.067190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.067154    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"default-dockercfg-h844f\""
Nov 02 12:36:37.067190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.067182    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"simulator-w4-843978-kserve-self-signed-certs\""
Nov 02 12:36:37.067427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.067340    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"kube-root-ca.crt\""
Nov 02 12:36:37.068581 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.068547    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"]
Nov 02 12:36:37.113298 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.113251    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:37.135506 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.135468    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-home\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.135506 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.135506    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-model-cache\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.135745 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.135529    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-dshm\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.135745 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.135553    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tls-certs\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.135745 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.135628    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tmp-dir\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.135850 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.135818    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mbzk\" (UniqueName: \"kubernetes.io/projected/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-kube-api-access-9mbzk\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.236746 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.236703    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-home\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.236746 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.236751    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-model-cache\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.237051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.236872    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-dshm\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.237051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.236912    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tls-certs\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.237051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.236942    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tmp-dir\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.237234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.237129    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mbzk\" (UniqueName: \"kubernetes.io/projected/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-kube-api-access-9mbzk\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.237296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.237251    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-home\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.237296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.237275    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-model-cache\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.237398 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.237331    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tmp-dir\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.239420 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.239398    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-dshm\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.239584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.239567    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tls-certs\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.263230 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.263198    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mbzk\" (UniqueName: \"kubernetes.io/projected/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-kube-api-access-9mbzk\") pod \"simulator-w4-843978-kserve-748d75d666-qzxl8\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") " pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.350025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.349899    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:37.657030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.656976    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"]
Nov 02 12:36:37.657509 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:36:37.657485    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac7edc2c_4ae5_4714_a9b6_1b07c6700083.slice/crio-d231650d012a85ebde453bb8c8b3a5d5c82e1da9838eae61998c30148d1da586 WatchSource:0}: Error finding container d231650d012a85ebde453bb8c8b3a5d5c82e1da9838eae61998c30148d1da586: Status 404 returned error can't find the container with id d231650d012a85ebde453bb8c8b3a5d5c82e1da9838eae61998c30148d1da586
Nov 02 12:36:37.777543 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.777500    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" event={"ID":"ac7edc2c-4ae5-4714-a9b6-1b07c6700083","Type":"ContainerStarted","Data":"89d66747eee021b156db4c9493412727e1b4b8cd138ae79c08212bf90c8fc42b"}
Nov 02 12:36:37.777723 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.777554    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" event={"ID":"ac7edc2c-4ae5-4714-a9b6-1b07c6700083","Type":"ContainerStarted","Data":"d231650d012a85ebde453bb8c8b3a5d5c82e1da9838eae61998c30148d1da586"}
Nov 02 12:36:37.822981 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:37.822924    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podStartSLOduration=1.822908103 podStartE2EDuration="1.822908103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:37.821842659 +0000 UTC m=+543.928545511" watchObservedRunningTime="2026-11-02 12:36:37.822908103 +0000 UTC m=+543.929610954"
Nov 02 12:36:38.053299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.053195    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:38.350461 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.350371    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:38.772789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.772756    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"]
Nov 02 12:36:38.776705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.776678    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.794620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.794590    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-068756\"/\"kube-root-ca.crt\""
Nov 02 12:36:38.808132 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.808092    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-068756\"/\"openshift-service-ca.crt\""
Nov 02 12:36:38.808307 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.808236    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-068756\"/\"default-dockercfg-99n5t\""
Nov 02 12:36:38.808394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.808335    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-068756\"/\"simulator-w3-068756-kserve-self-signed-certs\""
Nov 02 12:36:38.809912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.809889    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"]
Nov 02 12:36:38.856842 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.856797    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee92d4b-5a20-46ef-b035-85b87337be88-tls-certs\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.857063 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.856855    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsqxw\" (UniqueName: \"kubernetes.io/projected/bee92d4b-5a20-46ef-b035-85b87337be88-kube-api-access-qsqxw\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.857063 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.856913    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-model-cache\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.857174 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.857109    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-home\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.857174 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.857154    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-tmp-dir\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.857284 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.857188    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-dshm\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.870763 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.870723    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:38.958196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.958163    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee92d4b-5a20-46ef-b035-85b87337be88-tls-certs\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.958427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.958212    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsqxw\" (UniqueName: \"kubernetes.io/projected/bee92d4b-5a20-46ef-b035-85b87337be88-kube-api-access-qsqxw\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.958427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.958245    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-model-cache\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.958427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.958270    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-home\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.958427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.958294    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-tmp-dir\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.958427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.958314    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-dshm\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.958753 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.958733    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-model-cache\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.958807 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.958786    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-home\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.958848 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.958807    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-tmp-dir\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.960889 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.960864    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-dshm\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.961130 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.961113    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee92d4b-5a20-46ef-b035-85b87337be88-tls-certs\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:38.983640 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:38.983604    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsqxw\" (UniqueName: \"kubernetes.io/projected/bee92d4b-5a20-46ef-b035-85b87337be88-kube-api-access-qsqxw\") pod \"simulator-w3-068756-kserve-6969d4fc4b-bm6n4\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") " pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:39.092402 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:39.092304    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:39.395590 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:39.395559    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"]
Nov 02 12:36:39.396266 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:36:39.396237    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbee92d4b_5a20_46ef_b035_85b87337be88.slice/crio-63686d3f01185411b682d9e09dd780415db9546954524b6685977e08c1272045 WatchSource:0}: Error finding container 63686d3f01185411b682d9e09dd780415db9546954524b6685977e08c1272045: Status 404 returned error can't find the container with id 63686d3f01185411b682d9e09dd780415db9546954524b6685977e08c1272045
Nov 02 12:36:39.425816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:39.425779    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:39.438500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:39.438472    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:39.734789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:39.734707    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:39.794269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:39.794230    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" event={"ID":"bee92d4b-5a20-46ef-b035-85b87337be88","Type":"ContainerStarted","Data":"aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6"}
Nov 02 12:36:39.794496 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:39.794476    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" event={"ID":"bee92d4b-5a20-46ef-b035-85b87337be88","Type":"ContainerStarted","Data":"63686d3f01185411b682d9e09dd780415db9546954524b6685977e08c1272045"}
Nov 02 12:36:39.806594 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:39.806568    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:36:39.839212 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:39.839154    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podStartSLOduration=1.839139303 podStartE2EDuration="1.839139303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:39.838171419 +0000 UTC m=+545.944874271" watchObservedRunningTime="2026-11-02 12:36:39.839139303 +0000 UTC m=+545.945842155"
Nov 02 12:36:40.092733 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:40.092627    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:41.323539 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:41.323503    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:42.989159 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:42.989127    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:44.072209 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:44.072119    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:44.664104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:44.664061    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:44.821679 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:36:44.821643    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:36:46.255883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:46.255842    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:47.350619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:47.350580    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:47.363765 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:47.363733    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:47.847594 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:47.847567    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:36:47.976887 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:47.976854    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:48.212320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:48.212287    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:48.824571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:48.824538    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:48.952305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:48.952264    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:49.088099 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:49.088013    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:49.093036 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:49.093021    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:49.106050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:49.106028    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:49.676616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:49.676584    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:49.862427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:49.862399    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:36:51.392911 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:51.392871    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:52.916197 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:52.916158    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"]
Nov 02 12:36:52.920315 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:52.920291    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:52.941155 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:52.941118    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"]
Nov 02 12:36:52.951936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:52.951900    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:52.952291 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:52.952274    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-95ee26\"/\"premium-w2-95ee26-kserve-self-signed-certs\""
Nov 02 12:36:53.004059 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.004010    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tmp-dir\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.004245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.004155    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-dshm\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.004245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.004236    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-home\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.004387 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.004353    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tls-certs\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.004474 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.004450    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dltpn\" (UniqueName: \"kubernetes.io/projected/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-kube-api-access-dltpn\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.004536 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.004523    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-model-cache\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.082841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.082795    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:53.105548 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.105512    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-model-cache\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.105760 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.105558    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tmp-dir\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.105760 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.105615    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-dshm\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.105760 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.105671    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-home\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.105760 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.105752    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tls-certs\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.105966 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.105791    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dltpn\" (UniqueName: \"kubernetes.io/projected/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-kube-api-access-dltpn\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.106057 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.105971    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-model-cache\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.106446 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.106421    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-home\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.106606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.106577    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tmp-dir\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.109165 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.109134    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-dshm\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.109373 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.109358    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tls-certs\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.134573 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.134544    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dltpn\" (UniqueName: \"kubernetes.io/projected/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-kube-api-access-dltpn\") pod \"premium-w2-95ee26-kserve-5b9f48d64d-zkbvj\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") " pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.234096 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.233982    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:53.534253 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.534224    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"]
Nov 02 12:36:53.535553 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:36:53.535529    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6db32bc0_dcc9_4edf_9829_caf8a9ec4a70.slice/crio-2fab048514b6c3d47f354e21cc1ef472dd0a57dad19467c5b34f09697272e4c5 WatchSource:0}: Error finding container 2fab048514b6c3d47f354e21cc1ef472dd0a57dad19467c5b34f09697272e4c5: Status 404 returned error can't find the container with id 2fab048514b6c3d47f354e21cc1ef472dd0a57dad19467c5b34f09697272e4c5
Nov 02 12:36:53.873758 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.873722    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" event={"ID":"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70","Type":"ContainerStarted","Data":"7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee"}
Nov 02 12:36:53.873960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.873769    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" event={"ID":"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70","Type":"ContainerStarted","Data":"2fab048514b6c3d47f354e21cc1ef472dd0a57dad19467c5b34f09697272e4c5"}
Nov 02 12:36:53.918198 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:53.918142    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podStartSLOduration=1.9181282149999999 podStartE2EDuration="1.918128215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:36:53.917634822 +0000 UTC m=+560.024337667" watchObservedRunningTime="2026-11-02 12:36:53.918128215 +0000 UTC m=+560.024831066"
Nov 02 12:36:54.235033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:54.234938    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:36:54.647673 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:54.647627    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:56.248674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:56.248637    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:56.910500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:56.910464    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:57.869365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:57.869331    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:36:59.558214 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:36:59.558159    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:01.125568 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:01.125519    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:02.661714 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:02.661679    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:03.234755 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:03.234717    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:37:03.247764 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:03.247737    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:37:03.944981 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:03.944950    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:37:04.343902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:04.343816    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:05.988394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:05.988356    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:07.654393 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:07.654363    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:08.305609 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:08.305576    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:09.251936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:09.251838    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:10.941800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:10.941753    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:12.645804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:12.645766    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:14.451188 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:14.451119    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:14.821347 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:37:14.821261    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:37:15.153394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.153358    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"]
Nov 02 12:37:15.161932 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.161902    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.185399 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.185365    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"]
Nov 02 12:37:15.196583 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.196557    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-843978\"/\"premium-w4-843978-kserve-self-signed-certs\""
Nov 02 12:37:15.230245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.230205    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:15.256517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.256471    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5dlj\" (UniqueName: \"kubernetes.io/projected/03ad3fad-c135-4470-9d9d-429d0d2df2b4-kube-api-access-z5dlj\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.256736 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.256631    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-home\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.256736 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.256671    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tmp-dir\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.256736 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.256691    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-model-cache\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.256736 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.256711    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tls-certs\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.256960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.256871    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-dshm\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.358500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.358461    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-dshm\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.358730 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.358558    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5dlj\" (UniqueName: \"kubernetes.io/projected/03ad3fad-c135-4470-9d9d-429d0d2df2b4-kube-api-access-z5dlj\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.358730 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.358669    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-home\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.358730 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.358711    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tmp-dir\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.358932 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.358739    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-model-cache\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.358932 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.358768    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tls-certs\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.359351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.359317    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-home\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.359456 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.359421    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tmp-dir\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.359741 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.359655    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-model-cache\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.361710 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.361679    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-dshm\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.361937 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.361910    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tls-certs\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.385081 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.385045    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5dlj\" (UniqueName: \"kubernetes.io/projected/03ad3fad-c135-4470-9d9d-429d0d2df2b4-kube-api-access-z5dlj\") pod \"premium-w4-843978-kserve-58f57bc4c9-d7n7n\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") " pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.499468 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.499378    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:15.801982 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:15.801949    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"]
Nov 02 12:37:15.803351 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:37:15.803323    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03ad3fad_c135_4470_9d9d_429d0d2df2b4.slice/crio-9386e29414c22c2ad76ba3a56e6ea5470b9baadec896b9a0ca57a947d9b6f8dd WatchSource:0}: Error finding container 9386e29414c22c2ad76ba3a56e6ea5470b9baadec896b9a0ca57a947d9b6f8dd: Status 404 returned error can't find the container with id 9386e29414c22c2ad76ba3a56e6ea5470b9baadec896b9a0ca57a947d9b6f8dd
Nov 02 12:37:16.004051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:16.003983    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" event={"ID":"03ad3fad-c135-4470-9d9d-429d0d2df2b4","Type":"ContainerStarted","Data":"e7613b05a9a0d8f1e65e6f20cca3df8c9f0f8eb728d1c124cec4843a64b423f0"}
Nov 02 12:37:16.004229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:16.004067    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" event={"ID":"03ad3fad-c135-4470-9d9d-429d0d2df2b4","Type":"ContainerStarted","Data":"9386e29414c22c2ad76ba3a56e6ea5470b9baadec896b9a0ca57a947d9b6f8dd"}
Nov 02 12:37:16.051769 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:16.051670    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podStartSLOduration=1.051652761 podStartE2EDuration="1.051652761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:37:16.050276093 +0000 UTC m=+582.156978945" watchObservedRunningTime="2026-11-02 12:37:16.051652761 +0000 UTC m=+582.158355613"
Nov 02 12:37:16.060460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:16.060425    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:16.499765 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:16.499729    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:17.483684 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.483650    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"]
Nov 02 12:37:17.487860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.487833    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.518193 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.518162    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5514e5\"/\"openshift-service-ca.crt\""
Nov 02 12:37:17.519234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.519215    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5514e5\"/\"default-dockercfg-vbmk6\""
Nov 02 12:37:17.519344 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.519265    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5514e5\"/\"premium-w0-5514e5-kserve-self-signed-certs\""
Nov 02 12:37:17.520308 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.520286    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"]
Nov 02 12:37:17.520907 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.520888    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5514e5\"/\"kube-root-ca.crt\""
Nov 02 12:37:17.559648 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.559614    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:17.584682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.584635    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-model-cache\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.584884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.584691    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqtl\" (UniqueName: \"kubernetes.io/projected/171a46b7-75bc-4212-8cb7-3252b2001291-kube-api-access-9fqtl\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.584884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.584839    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-dshm\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.585028 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.584887    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-tmp-dir\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.585028 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.584980    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/171a46b7-75bc-4212-8cb7-3252b2001291-tls-certs\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.585145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.585044    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-home\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.686291 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.686240    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:17.686900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.686871    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqtl\" (UniqueName: \"kubernetes.io/projected/171a46b7-75bc-4212-8cb7-3252b2001291-kube-api-access-9fqtl\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.687087 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.686978    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-dshm\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.687087 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.687078    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-tmp-dir\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.687209 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.687128    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/171a46b7-75bc-4212-8cb7-3252b2001291-tls-certs\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.687258 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.687236    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-home\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.687340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.687324    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-model-cache\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.687729 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.687706    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-tmp-dir\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.687830 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.687745    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-model-cache\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.688161 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.688111    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-home\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.691035 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.691010    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/171a46b7-75bc-4212-8cb7-3252b2001291-tls-certs\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.691323 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.691296    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-dshm\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.712669 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.712637    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqtl\" (UniqueName: \"kubernetes.io/projected/171a46b7-75bc-4212-8cb7-3252b2001291-kube-api-access-9fqtl\") pod \"premium-w0-5514e5-kserve-785c9dd85c-fqkgv\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") " pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:17.802791 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:17.802695    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:18.106145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:18.106114    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"]
Nov 02 12:37:18.106964 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:37:18.106936    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod171a46b7_75bc_4212_8cb7_3252b2001291.slice/crio-3c24b08047dbaf87edbbeaa36418e6660b2025268efe7c840ab9c2140c1cd93f WatchSource:0}: Error finding container 3c24b08047dbaf87edbbeaa36418e6660b2025268efe7c840ab9c2140c1cd93f: Status 404 returned error can't find the container with id 3c24b08047dbaf87edbbeaa36418e6660b2025268efe7c840ab9c2140c1cd93f
Nov 02 12:37:18.314848 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:18.314806    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:19.028477 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:19.028437    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" event={"ID":"171a46b7-75bc-4212-8cb7-3252b2001291","Type":"ContainerStarted","Data":"a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d"}
Nov 02 12:37:19.028882 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:19.028494    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" event={"ID":"171a46b7-75bc-4212-8cb7-3252b2001291","Type":"ContainerStarted","Data":"3c24b08047dbaf87edbbeaa36418e6660b2025268efe7c840ab9c2140c1cd93f"}
Nov 02 12:37:19.074585 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:19.074528    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podStartSLOduration=2.0745128250000002 podStartE2EDuration="2.074512825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:37:19.074027851 +0000 UTC m=+585.180730700" watchObservedRunningTime="2026-11-02 12:37:19.074512825 +0000 UTC m=+585.181215672"
Nov 02 12:37:19.334793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:19.334689    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:19.803135 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:19.803096    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:20.945047 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:20.944987    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:22.649458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:22.649419    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:24.334252 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:24.334220    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:25.086360 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:25.086320    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:25.499602 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:25.499565    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:25.512823 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:25.512795    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:25.961062 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:25.961026    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:26.089567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:26.089537    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:37:27.547288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:27.547251    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:27.803865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:27.803775    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:27.817278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:27.817240    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:28.100507 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:28.100428    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:37:29.147744 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:29.147707    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:30.708705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:30.708665    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:32.468425 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:32.468384    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:33.122153 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.122112    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"]
Nov 02 12:37:33.126419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.126395    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:33.143631 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.143583    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:37:33.145772 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.145739    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"]
Nov 02 12:37:33.157086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.157061    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:37:33.253268 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.253222    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e781860f-9hpzs\" (UID: \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:33.253444 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.253284    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xknn7\" (UniqueName: \"kubernetes.io/projected/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-kube-api-access-xknn7\") pod \"maas-api-revoke-sub-e781860f-9hpzs\" (UID: \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:33.253578 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.253554    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e781860f-9hpzs\" (UID: \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:33.354917 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.354881    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e781860f-9hpzs\" (UID: \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:33.355129 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.354936    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e781860f-9hpzs\" (UID: \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:33.355129 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.354985    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknn7\" (UniqueName: \"kubernetes.io/projected/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-kube-api-access-xknn7\") pod \"maas-api-revoke-sub-e781860f-9hpzs\" (UID: \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:33.355727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.355702    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e781860f-9hpzs\" (UID: \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:33.357749 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.357728    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e781860f-9hpzs\" (UID: \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:33.379931 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.379864    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xknn7\" (UniqueName: \"kubernetes.io/projected/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-kube-api-access-xknn7\") pod \"maas-api-revoke-sub-e781860f-9hpzs\" (UID: \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:33.442145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.442105    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:33.750392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.750357    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"]
Nov 02 12:37:33.751712 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:37:33.751687    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efe6021_b363_4f36_bacb_0cb4ac8c0b47.slice/crio-ba4f8d196a0fdc6dbd1568c052c0ea109a42a8bddadbd2f8d4857798d2a9ff49 WatchSource:0}: Error finding container ba4f8d196a0fdc6dbd1568c052c0ea109a42a8bddadbd2f8d4857798d2a9ff49: Status 404 returned error can't find the container with id ba4f8d196a0fdc6dbd1568c052c0ea109a42a8bddadbd2f8d4857798d2a9ff49
Nov 02 12:37:33.768704 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:33.768674    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:34.043174 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:34.043100    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:34.121013 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:34.120953    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs" event={"ID":"5efe6021-b363-4f36-bacb-0cb4ac8c0b47","Type":"ContainerStarted","Data":"1d0de91759bd46fb302012294bb96b8e6f9f452e838d43748cc6425fdcfe28dd"}
Nov 02 12:37:34.121208 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:34.121093    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs" event={"ID":"5efe6021-b363-4f36-bacb-0cb4ac8c0b47","Type":"ContainerStarted","Data":"ba4f8d196a0fdc6dbd1568c052c0ea109a42a8bddadbd2f8d4857798d2a9ff49"}
Nov 02 12:37:34.164189 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:34.164135    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs" podStartSLOduration=1.164119098 podStartE2EDuration="1.164119098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:37:34.16251096 +0000 UTC m=+600.269213812" watchObservedRunningTime="2026-11-02 12:37:34.164119098 +0000 UTC m=+600.270821950"
Nov 02 12:37:35.128906 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:35.128868    5879 generic.go:356] "Generic (PLEG): container finished" podID="5efe6021-b363-4f36-bacb-0cb4ac8c0b47" containerID="1d0de91759bd46fb302012294bb96b8e6f9f452e838d43748cc6425fdcfe28dd" exitCode=0
Nov 02 12:37:35.129376 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:35.128953    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs" event={"ID":"5efe6021-b363-4f36-bacb-0cb4ac8c0b47","Type":"ContainerDied","Data":"1d0de91759bd46fb302012294bb96b8e6f9f452e838d43748cc6425fdcfe28dd"}
Nov 02 12:37:35.642626 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:35.642584    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:36.276810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:36.276779    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:36.387264 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:36.387181    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-xks-ca\") pod \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\" (UID: \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\") "
Nov 02 12:37:36.387426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:36.387288    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-openshift-ca\") pod \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\" (UID: \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\") "
Nov 02 12:37:36.387426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:36.387413    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-kube-api-access-xknn7\" (UniqueName: \"kubernetes.io/projected/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-kube-api-access-xknn7\") pod \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\" (UID: \"5efe6021-b363-4f36-bacb-0cb4ac8c0b47\") "
Nov 02 12:37:36.387679 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:36.387650    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-openshift-ca" pod "5efe6021-b363-4f36-bacb-0cb4ac8c0b47" (UID: "5efe6021-b363-4f36-bacb-0cb4ac8c0b47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:37:36.389600 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:36.389575    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-xks-ca" pod "5efe6021-b363-4f36-bacb-0cb4ac8c0b47" (UID: "5efe6021-b363-4f36-bacb-0cb4ac8c0b47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:37:36.389769 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:36.389747    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-kube-api-access-xknn7" pod "5efe6021-b363-4f36-bacb-0cb4ac8c0b47" (UID: "5efe6021-b363-4f36-bacb-0cb4ac8c0b47"). InnerVolumeSpecName "kube-api-access-xknn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:37:36.488915 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:36.488878    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xknn7\" (UniqueName: \"kubernetes.io/projected/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-kube-api-access-xknn7\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:37:36.488915 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:36.488916    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:37:36.488915 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:36.488931    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5efe6021-b363-4f36-bacb-0cb4ac8c0b47-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:37:37.141469 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:37.141432    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"
Nov 02 12:37:37.141659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:37.141474    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs" event={"ID":"5efe6021-b363-4f36-bacb-0cb4ac8c0b47","Type":"ContainerDied","Data":"ba4f8d196a0fdc6dbd1568c052c0ea109a42a8bddadbd2f8d4857798d2a9ff49"}
Nov 02 12:37:37.141659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:37.141530    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba4f8d196a0fdc6dbd1568c052c0ea109a42a8bddadbd2f8d4857798d2a9ff49"
Nov 02 12:37:37.357741 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:37.357704    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:38.954723 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:38.954687    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:40.660341 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:40.660304    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:42.434387 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:42.434350    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:44.152588 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:44.152546    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:44.821277 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:37:44.821200    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:37:45.655008 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:45.654962    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:47.216374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:47.216331    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:48.379511 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.379473    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"]
Nov 02 12:37:48.380021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.379983    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5efe6021-b363-4f36-bacb-0cb4ac8c0b47" containerName="revoke-keys"
Nov 02 12:37:48.380021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.380019    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5efe6021-b363-4f36-bacb-0cb4ac8c0b47" containerName="revoke-keys"
Nov 02 12:37:48.380133 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.380123    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5efe6021-b363-4f36-bacb-0cb4ac8c0b47" containerName="revoke-keys"
Nov 02 12:37:48.383682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.383666    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:48.400869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.400835    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:37:48.403228 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.403195    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"]
Nov 02 12:37:48.414516 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.414489    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:37:48.524897 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.524855    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4606331a-zj89d\" (UID: \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:48.525076 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.524987    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4606331a-zj89d\" (UID: \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:48.525139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.525116    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8jls\" (UniqueName: \"kubernetes.io/projected/33290e89-7f9a-4899-9c57-8b4231b2dd5f-kube-api-access-n8jls\") pod \"maas-api-revoke-sub-4606331a-zj89d\" (UID: \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:48.626456 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.626413    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8jls\" (UniqueName: \"kubernetes.io/projected/33290e89-7f9a-4899-9c57-8b4231b2dd5f-kube-api-access-n8jls\") pod \"maas-api-revoke-sub-4606331a-zj89d\" (UID: \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:48.626646 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.626476    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4606331a-zj89d\" (UID: \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:48.626646 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.626537    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4606331a-zj89d\" (UID: \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:48.627360 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.627336    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4606331a-zj89d\" (UID: \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:48.629210 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.629186    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4606331a-zj89d\" (UID: \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:48.651674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.651611    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8jls\" (UniqueName: \"kubernetes.io/projected/33290e89-7f9a-4899-9c57-8b4231b2dd5f-kube-api-access-n8jls\") pod \"maas-api-revoke-sub-4606331a-zj89d\" (UID: \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:48.697599 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.697567    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:48.750325 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:48.750290    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:49.001497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:49.001468    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"]
Nov 02 12:37:49.002703 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:37:49.002679    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33290e89_7f9a_4899_9c57_8b4231b2dd5f.slice/crio-c6eda6795724f03f0f3f9349a643fda3bcf58d4051557cfe98c0766bce18f7c3 WatchSource:0}: Error finding container c6eda6795724f03f0f3f9349a643fda3bcf58d4051557cfe98c0766bce18f7c3: Status 404 returned error can't find the container with id c6eda6795724f03f0f3f9349a643fda3bcf58d4051557cfe98c0766bce18f7c3
Nov 02 12:37:49.206314 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:49.206278    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d" event={"ID":"33290e89-7f9a-4899-9c57-8b4231b2dd5f","Type":"ContainerStarted","Data":"c6eda6795724f03f0f3f9349a643fda3bcf58d4051557cfe98c0766bce18f7c3"}
Nov 02 12:37:50.213354 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:50.213256    5879 generic.go:356] "Generic (PLEG): container finished" podID="33290e89-7f9a-4899-9c57-8b4231b2dd5f" containerID="5b284d5e01a038bdebc6a5f48998b5cbc1f94e3218c093560965b93a4c5da84d" exitCode=0
Nov 02 12:37:50.213748 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:50.213340    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d" event={"ID":"33290e89-7f9a-4899-9c57-8b4231b2dd5f","Type":"ContainerDied","Data":"5b284d5e01a038bdebc6a5f48998b5cbc1f94e3218c093560965b93a4c5da84d"}
Nov 02 12:37:50.351183 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:50.351149    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:51.374235 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:51.374210    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:51.558406 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:51.558300    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33290e89-7f9a-4899-9c57-8b4231b2dd5f-kube-api-access-n8jls\" (UniqueName: \"kubernetes.io/projected/33290e89-7f9a-4899-9c57-8b4231b2dd5f-kube-api-access-n8jls\") pod \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\" (UID: \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\") "
Nov 02 12:37:51.558406 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:51.558360    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-openshift-ca\") pod \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\" (UID: \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\") "
Nov 02 12:37:51.558662 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:51.558485    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-xks-ca\") pod \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\" (UID: \"33290e89-7f9a-4899-9c57-8b4231b2dd5f\") "
Nov 02 12:37:51.558779 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:51.558754    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-openshift-ca" pod "33290e89-7f9a-4899-9c57-8b4231b2dd5f" (UID: "33290e89-7f9a-4899-9c57-8b4231b2dd5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:37:51.559070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:51.559049    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:37:51.560812 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:51.560785    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33290e89-7f9a-4899-9c57-8b4231b2dd5f-kube-api-access-n8jls" pod "33290e89-7f9a-4899-9c57-8b4231b2dd5f" (UID: "33290e89-7f9a-4899-9c57-8b4231b2dd5f"). InnerVolumeSpecName "kube-api-access-n8jls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:37:51.560949 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:51.560841    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-xks-ca" pod "33290e89-7f9a-4899-9c57-8b4231b2dd5f" (UID: "33290e89-7f9a-4899-9c57-8b4231b2dd5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:37:51.659575 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:51.659538    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8jls\" (UniqueName: \"kubernetes.io/projected/33290e89-7f9a-4899-9c57-8b4231b2dd5f-kube-api-access-n8jls\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:37:51.659575 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:51.659572    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33290e89-7f9a-4899-9c57-8b4231b2dd5f-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:37:52.126708 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:52.126677    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:52.228609 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:52.228573    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"
Nov 02 12:37:52.228789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:52.228606    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d" event={"ID":"33290e89-7f9a-4899-9c57-8b4231b2dd5f","Type":"ContainerDied","Data":"c6eda6795724f03f0f3f9349a643fda3bcf58d4051557cfe98c0766bce18f7c3"}
Nov 02 12:37:52.228789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:52.228670    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6eda6795724f03f0f3f9349a643fda3bcf58d4051557cfe98c0766bce18f7c3"
Nov 02 12:37:53.858223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:53.858179    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:55.463184 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:55.463150    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:56.388741 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.388711    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"]
Nov 02 12:37:56.389224 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.389209    5879 cpu_manager.go:414] "containerMap: removing container" podUID="33290e89-7f9a-4899-9c57-8b4231b2dd5f" containerName="revoke-keys"
Nov 02 12:37:56.389224 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.389224    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33290e89-7f9a-4899-9c57-8b4231b2dd5f" containerName="revoke-keys"
Nov 02 12:37:56.389373 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.389317    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="33290e89-7f9a-4899-9c57-8b4231b2dd5f" containerName="revoke-keys"
Nov 02 12:37:56.394126 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.394094    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:37:56.411090 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.411054    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tr97\" (UniqueName: \"kubernetes.io/projected/2d031a40-16b1-4f22-8a9f-216404661f78-kube-api-access-2tr97\") pod \"maas-api-revoke-sub-bbacaaa9-cjrdh\" (UID: \"2d031a40-16b1-4f22-8a9f-216404661f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:37:56.411299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.411123    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbacaaa9-cjrdh\" (UID: \"2d031a40-16b1-4f22-8a9f-216404661f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:37:56.411299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.411180    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbacaaa9-cjrdh\" (UID: \"2d031a40-16b1-4f22-8a9f-216404661f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:37:56.412538 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.412516    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"]
Nov 02 12:37:56.423509 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.423473    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:37:56.423680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.423534    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:37:56.512833 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.512801    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tr97\" (UniqueName: \"kubernetes.io/projected/2d031a40-16b1-4f22-8a9f-216404661f78-kube-api-access-2tr97\") pod \"maas-api-revoke-sub-bbacaaa9-cjrdh\" (UID: \"2d031a40-16b1-4f22-8a9f-216404661f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:37:56.513269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.512851    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbacaaa9-cjrdh\" (UID: \"2d031a40-16b1-4f22-8a9f-216404661f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:37:56.513269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.512900    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbacaaa9-cjrdh\" (UID: \"2d031a40-16b1-4f22-8a9f-216404661f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:37:56.513688 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.513671    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbacaaa9-cjrdh\" (UID: \"2d031a40-16b1-4f22-8a9f-216404661f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:37:56.515458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.515439    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbacaaa9-cjrdh\" (UID: \"2d031a40-16b1-4f22-8a9f-216404661f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:37:56.538054 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.538022    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tr97\" (UniqueName: \"kubernetes.io/projected/2d031a40-16b1-4f22-8a9f-216404661f78-kube-api-access-2tr97\") pod \"maas-api-revoke-sub-bbacaaa9-cjrdh\" (UID: \"2d031a40-16b1-4f22-8a9f-216404661f78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:37:56.617513 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.617476    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"]
Nov 02 12:37:56.621579 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.621557    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:37:56.641004 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.640887    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"]
Nov 02 12:37:56.708442 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.708408    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:37:56.718225 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.715970    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7283989-l8psw\" (UID: \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:37:56.718225 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.716039    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5jdp\" (UniqueName: \"kubernetes.io/projected/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-kube-api-access-m5jdp\") pod \"maas-api-revoke-sub-b7283989-l8psw\" (UID: \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:37:56.718225 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.716161    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7283989-l8psw\" (UID: \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:37:56.817723 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.817629    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7283989-l8psw\" (UID: \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:37:56.817723 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.817681    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5jdp\" (UniqueName: \"kubernetes.io/projected/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-kube-api-access-m5jdp\") pod \"maas-api-revoke-sub-b7283989-l8psw\" (UID: \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:37:56.817944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.817818    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7283989-l8psw\" (UID: \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:37:56.818292 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.818266    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7283989-l8psw\" (UID: \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:37:56.820560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.820535    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7283989-l8psw\" (UID: \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:37:56.843171 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.843135    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5jdp\" (UniqueName: \"kubernetes.io/projected/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-kube-api-access-m5jdp\") pod \"maas-api-revoke-sub-b7283989-l8psw\" (UID: \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:37:56.935550 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:56.935462    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:37:57.015195 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:57.015157    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"]
Nov 02 12:37:57.015824 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:37:57.015797    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d031a40_16b1_4f22_8a9f_216404661f78.slice/crio-d4deb76e2df444a598d5fef462dd2fc053c3fadd66bdb8e2a0fca7e7905f138d WatchSource:0}: Error finding container d4deb76e2df444a598d5fef462dd2fc053c3fadd66bdb8e2a0fca7e7905f138d: Status 404 returned error can't find the container with id d4deb76e2df444a598d5fef462dd2fc053c3fadd66bdb8e2a0fca7e7905f138d
Nov 02 12:37:57.146923 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:57.146884    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:57.237053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:57.237025    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"]
Nov 02 12:37:57.237553 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:37:57.237513    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0c96d2_31a7_44a6_b3ec_7544395a47aa.slice/crio-378dd03af476e3e5f5e29dba6f1f718064a9abdbca731c088a5f60ae67afdea6 WatchSource:0}: Error finding container 378dd03af476e3e5f5e29dba6f1f718064a9abdbca731c088a5f60ae67afdea6: Status 404 returned error can't find the container with id 378dd03af476e3e5f5e29dba6f1f718064a9abdbca731c088a5f60ae67afdea6
Nov 02 12:37:57.256043 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:57.255973    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw" event={"ID":"7f0c96d2-31a7-44a6-b3ec-7544395a47aa","Type":"ContainerStarted","Data":"378dd03af476e3e5f5e29dba6f1f718064a9abdbca731c088a5f60ae67afdea6"}
Nov 02 12:37:57.257395 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:57.257367    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh" event={"ID":"2d031a40-16b1-4f22-8a9f-216404661f78","Type":"ContainerStarted","Data":"d4deb76e2df444a598d5fef462dd2fc053c3fadd66bdb8e2a0fca7e7905f138d"}
Nov 02 12:37:58.273498 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:58.273399    5879 generic.go:356] "Generic (PLEG): container finished" podID="7f0c96d2-31a7-44a6-b3ec-7544395a47aa" containerID="3591cf5b44e3f73e597cefb7314d0374847032bb0dcb68384418504a3a5d23dc" exitCode=0
Nov 02 12:37:58.273498 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:58.273482    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw" event={"ID":"7f0c96d2-31a7-44a6-b3ec-7544395a47aa","Type":"ContainerDied","Data":"3591cf5b44e3f73e597cefb7314d0374847032bb0dcb68384418504a3a5d23dc"}
Nov 02 12:37:58.275179 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:58.275153    5879 generic.go:356] "Generic (PLEG): container finished" podID="2d031a40-16b1-4f22-8a9f-216404661f78" containerID="ca03b8de5f5057ed5e3af948cbb1553a780ac8b92ddbbc83ab5ab4423e73f970" exitCode=0
Nov 02 12:37:58.275289 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:58.275238    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh" event={"ID":"2d031a40-16b1-4f22-8a9f-216404661f78","Type":"ContainerDied","Data":"ca03b8de5f5057ed5e3af948cbb1553a780ac8b92ddbbc83ab5ab4423e73f970"}
Nov 02 12:37:58.688568 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:58.688532    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:37:59.446725 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.446694    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:37:59.450011 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.449978    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:37:59.548613 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.548537    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d031a40-16b1-4f22-8a9f-216404661f78-kube-api-access-2tr97\" (UniqueName: \"kubernetes.io/projected/2d031a40-16b1-4f22-8a9f-216404661f78-kube-api-access-2tr97\") pod \"2d031a40-16b1-4f22-8a9f-216404661f78\" (UID: \"2d031a40-16b1-4f22-8a9f-216404661f78\") "
Nov 02 12:37:59.548613 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.548580    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-kube-api-access-m5jdp\" (UniqueName: \"kubernetes.io/projected/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-kube-api-access-m5jdp\") pod \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\" (UID: \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\") "
Nov 02 12:37:59.548823 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.548645    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-xks-ca\") pod \"2d031a40-16b1-4f22-8a9f-216404661f78\" (UID: \"2d031a40-16b1-4f22-8a9f-216404661f78\") "
Nov 02 12:37:59.548823 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.548691    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-openshift-ca\") pod \"2d031a40-16b1-4f22-8a9f-216404661f78\" (UID: \"2d031a40-16b1-4f22-8a9f-216404661f78\") "
Nov 02 12:37:59.548823 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.548792    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-openshift-ca\") pod \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\" (UID: \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\") "
Nov 02 12:37:59.549027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.548859    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-xks-ca\") pod \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\" (UID: \"7f0c96d2-31a7-44a6-b3ec-7544395a47aa\") "
Nov 02 12:37:59.549556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.549183    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-openshift-ca" pod "2d031a40-16b1-4f22-8a9f-216404661f78" (UID: "2d031a40-16b1-4f22-8a9f-216404661f78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:37:59.549556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.549474    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-openshift-ca" pod "7f0c96d2-31a7-44a6-b3ec-7544395a47aa" (UID: "7f0c96d2-31a7-44a6-b3ec-7544395a47aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:37:59.549730 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.549652    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:37:59.549730 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.549673    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:37:59.551890 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.551849    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-xks-ca" pod "7f0c96d2-31a7-44a6-b3ec-7544395a47aa" (UID: "7f0c96d2-31a7-44a6-b3ec-7544395a47aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:37:59.552034 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.551914    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-xks-ca" pod "2d031a40-16b1-4f22-8a9f-216404661f78" (UID: "2d031a40-16b1-4f22-8a9f-216404661f78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:37:59.552108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.552087    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-kube-api-access-m5jdp" pod "7f0c96d2-31a7-44a6-b3ec-7544395a47aa" (UID: "7f0c96d2-31a7-44a6-b3ec-7544395a47aa"). InnerVolumeSpecName "kube-api-access-m5jdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:37:59.552248 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.552230    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d031a40-16b1-4f22-8a9f-216404661f78-kube-api-access-2tr97" pod "2d031a40-16b1-4f22-8a9f-216404661f78" (UID: "2d031a40-16b1-4f22-8a9f-216404661f78"). InnerVolumeSpecName "kube-api-access-2tr97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:37:59.650910 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.650871    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tr97\" (UniqueName: \"kubernetes.io/projected/2d031a40-16b1-4f22-8a9f-216404661f78-kube-api-access-2tr97\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:37:59.650910 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.650903    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5jdp\" (UniqueName: \"kubernetes.io/projected/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-kube-api-access-m5jdp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:37:59.650910 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.650913    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d031a40-16b1-4f22-8a9f-216404661f78-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:37:59.650910 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.650922    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f0c96d2-31a7-44a6-b3ec-7544395a47aa-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:37:59.782650 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.782609    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"]
Nov 02 12:37:59.783252 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.783216    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2d031a40-16b1-4f22-8a9f-216404661f78" containerName="revoke-keys"
Nov 02 12:37:59.783377 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.783276    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d031a40-16b1-4f22-8a9f-216404661f78" containerName="revoke-keys"
Nov 02 12:37:59.783377 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.783292    5879 cpu_manager.go:414] "containerMap: removing container" podUID="7f0c96d2-31a7-44a6-b3ec-7544395a47aa" containerName="revoke-keys"
Nov 02 12:37:59.783377 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.783299    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0c96d2-31a7-44a6-b3ec-7544395a47aa" containerName="revoke-keys"
Nov 02 12:37:59.783533 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.783506    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d031a40-16b1-4f22-8a9f-216404661f78" containerName="revoke-keys"
Nov 02 12:37:59.783533 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.783520    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0c96d2-31a7-44a6-b3ec-7544395a47aa" containerName="revoke-keys"
Nov 02 12:37:59.787727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.787701    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:37:59.808612 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.808521    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"]
Nov 02 12:37:59.852302 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.852270    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39a9d2ca-r7ztn\" (UID: \"8175b87e-e393-44de-9a73-66b76e696d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:37:59.852494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.852340    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39a9d2ca-r7ztn\" (UID: \"8175b87e-e393-44de-9a73-66b76e696d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:37:59.852544 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.852503    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxnb\" (UniqueName: \"kubernetes.io/projected/8175b87e-e393-44de-9a73-66b76e696d50-kube-api-access-7zxnb\") pod \"maas-api-revoke-sub-39a9d2ca-r7ztn\" (UID: \"8175b87e-e393-44de-9a73-66b76e696d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:37:59.953136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.953102    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39a9d2ca-r7ztn\" (UID: \"8175b87e-e393-44de-9a73-66b76e696d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:37:59.953317 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.953179    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxnb\" (UniqueName: \"kubernetes.io/projected/8175b87e-e393-44de-9a73-66b76e696d50-kube-api-access-7zxnb\") pod \"maas-api-revoke-sub-39a9d2ca-r7ztn\" (UID: \"8175b87e-e393-44de-9a73-66b76e696d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:37:59.953375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.953312    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39a9d2ca-r7ztn\" (UID: \"8175b87e-e393-44de-9a73-66b76e696d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:37:59.953961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.953935    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39a9d2ca-r7ztn\" (UID: \"8175b87e-e393-44de-9a73-66b76e696d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:37:59.955680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.955655    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39a9d2ca-r7ztn\" (UID: \"8175b87e-e393-44de-9a73-66b76e696d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:37:59.978366 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:37:59.978335    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxnb\" (UniqueName: \"kubernetes.io/projected/8175b87e-e393-44de-9a73-66b76e696d50-kube-api-access-7zxnb\") pod \"maas-api-revoke-sub-39a9d2ca-r7ztn\" (UID: \"8175b87e-e393-44de-9a73-66b76e696d50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:38:00.102445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:00.102362    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:38:00.260882 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:00.260848    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:00.289782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:00.289748    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"
Nov 02 12:38:00.289782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:00.289763    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh" event={"ID":"2d031a40-16b1-4f22-8a9f-216404661f78","Type":"ContainerDied","Data":"d4deb76e2df444a598d5fef462dd2fc053c3fadd66bdb8e2a0fca7e7905f138d"}
Nov 02 12:38:00.290079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:00.289826    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4deb76e2df444a598d5fef462dd2fc053c3fadd66bdb8e2a0fca7e7905f138d"
Nov 02 12:38:00.291440 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:00.291416    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"
Nov 02 12:38:00.291440 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:00.291425    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw" event={"ID":"7f0c96d2-31a7-44a6-b3ec-7544395a47aa","Type":"ContainerDied","Data":"378dd03af476e3e5f5e29dba6f1f718064a9abdbca731c088a5f60ae67afdea6"}
Nov 02 12:38:00.291644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:00.291463    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="378dd03af476e3e5f5e29dba6f1f718064a9abdbca731c088a5f60ae67afdea6"
Nov 02 12:38:00.404136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:00.404107    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"]
Nov 02 12:38:00.404869 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:38:00.404818    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8175b87e_e393_44de_9a73_66b76e696d50.slice/crio-862abc26d2d35b46402c84825477244d7a67a0248fd838293829f49040f3f4d6 WatchSource:0}: Error finding container 862abc26d2d35b46402c84825477244d7a67a0248fd838293829f49040f3f4d6: Status 404 returned error can't find the container with id 862abc26d2d35b46402c84825477244d7a67a0248fd838293829f49040f3f4d6
Nov 02 12:38:01.297962 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.297929    5879 generic.go:356] "Generic (PLEG): container finished" podID="8175b87e-e393-44de-9a73-66b76e696d50" containerID="51d5f0ecbf1eec4f01aa93fc3abbf811663274dff4d4ff061b59fa54a461aeaa" exitCode=0
Nov 02 12:38:01.298362 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.298015    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn" event={"ID":"8175b87e-e393-44de-9a73-66b76e696d50","Type":"ContainerDied","Data":"51d5f0ecbf1eec4f01aa93fc3abbf811663274dff4d4ff061b59fa54a461aeaa"}
Nov 02 12:38:01.298362 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.298058    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn" event={"ID":"8175b87e-e393-44de-9a73-66b76e696d50","Type":"ContainerStarted","Data":"862abc26d2d35b46402c84825477244d7a67a0248fd838293829f49040f3f4d6"}
Nov 02 12:38:01.385883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.385842    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"]
Nov 02 12:38:01.389693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.389670    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.411065 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.411036    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-843978-openshift-default-dockercfg-nkwwj\""
Nov 02 12:38:01.413232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.413209    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"]
Nov 02 12:38:01.470786 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.470702    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.470786 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.470758    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.470979 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.470836    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.470979 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.470885    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.470979 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.470950    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.471103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.471033    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9thv5\" (UniqueName: \"kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-kube-api-access-9thv5\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.471103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.471063    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1816f3b7-b324-45cd-add2-9931d7f1eb26-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.471103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.471090    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.471214 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.471191    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.572015 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.571954    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.572207 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572050    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.572207 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572072    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.572207 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572103    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.572207 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572159    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.572207 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572196    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9thv5\" (UniqueName: \"kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-kube-api-access-9thv5\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.572472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572220    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1816f3b7-b324-45cd-add2-9931d7f1eb26-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.572472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572244    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.572472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572283    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.572472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572396    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.572655 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572586    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.572765 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572732    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.573006 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572774    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.573006 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.572911    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1816f3b7-b324-45cd-add2-9931d7f1eb26-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.574622 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.574601    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.574948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.574926    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.597745 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.597706    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.597745 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.597723    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9thv5\" (UniqueName: \"kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-kube-api-access-9thv5\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.702909 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.702864    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:01.954040 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:01.953968    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:02.025174 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.025143    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"]
Nov 02 12:38:02.025771 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:38:02.025740    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1816f3b7_b324_45cd_add2_9931d7f1eb26.slice/crio-2b40c32a5ad85f8233bcbf09212a59d77d04a27d9514d5298f501a97934df3f7 WatchSource:0}: Error finding container 2b40c32a5ad85f8233bcbf09212a59d77d04a27d9514d5298f501a97934df3f7: Status 404 returned error can't find the container with id 2b40c32a5ad85f8233bcbf09212a59d77d04a27d9514d5298f501a97934df3f7
Nov 02 12:38:02.028365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.028326    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:02.028473 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.028413    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:02.028473 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.028459    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:02.306375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.306285    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w" event={"ID":"1816f3b7-b324-45cd-add2-9931d7f1eb26","Type":"ContainerStarted","Data":"6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47"}
Nov 02 12:38:02.306771 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.306384    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w" event={"ID":"1816f3b7-b324-45cd-add2-9931d7f1eb26","Type":"ContainerStarted","Data":"2b40c32a5ad85f8233bcbf09212a59d77d04a27d9514d5298f501a97934df3f7"}
Nov 02 12:38:02.356553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.356499    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w" podStartSLOduration=1.356481522 podStartE2EDuration="1.356481522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:02.353676735 +0000 UTC m=+628.460379585" watchObservedRunningTime="2026-11-02 12:38:02.356481522 +0000 UTC m=+628.463184372"
Nov 02 12:38:02.445324 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.445295    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:38:02.607494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.606635    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-openshift-ca\") pod \"8175b87e-e393-44de-9a73-66b76e696d50\" (UID: \"8175b87e-e393-44de-9a73-66b76e696d50\") "
Nov 02 12:38:02.607494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.606714    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8175b87e-e393-44de-9a73-66b76e696d50-kube-api-access-7zxnb\" (UniqueName: \"kubernetes.io/projected/8175b87e-e393-44de-9a73-66b76e696d50-kube-api-access-7zxnb\") pod \"8175b87e-e393-44de-9a73-66b76e696d50\" (UID: \"8175b87e-e393-44de-9a73-66b76e696d50\") "
Nov 02 12:38:02.607494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.606761    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-xks-ca\") pod \"8175b87e-e393-44de-9a73-66b76e696d50\" (UID: \"8175b87e-e393-44de-9a73-66b76e696d50\") "
Nov 02 12:38:02.607494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.607198    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-openshift-ca" pod "8175b87e-e393-44de-9a73-66b76e696d50" (UID: "8175b87e-e393-44de-9a73-66b76e696d50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:02.607494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.607370    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:02.610616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.610575    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8175b87e-e393-44de-9a73-66b76e696d50-kube-api-access-7zxnb" pod "8175b87e-e393-44de-9a73-66b76e696d50" (UID: "8175b87e-e393-44de-9a73-66b76e696d50"). InnerVolumeSpecName "kube-api-access-7zxnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:02.611449 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.611388    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-xks-ca" pod "8175b87e-e393-44de-9a73-66b76e696d50" (UID: "8175b87e-e393-44de-9a73-66b76e696d50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:38:02.704019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.703106    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:02.708473 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.708444    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zxnb\" (UniqueName: \"kubernetes.io/projected/8175b87e-e393-44de-9a73-66b76e696d50-kube-api-access-7zxnb\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:02.708473 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:02.708471    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8175b87e-e393-44de-9a73-66b76e696d50-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:03.313481 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:03.313452    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"
Nov 02 12:38:03.313882 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:03.313477    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn" event={"ID":"8175b87e-e393-44de-9a73-66b76e696d50","Type":"ContainerDied","Data":"862abc26d2d35b46402c84825477244d7a67a0248fd838293829f49040f3f4d6"}
Nov 02 12:38:03.313882 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:03.313533    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="862abc26d2d35b46402c84825477244d7a67a0248fd838293829f49040f3f4d6"
Nov 02 12:38:03.669896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:03.669838    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:03.707354 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:03.707309    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:38:03.707560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:03.707390    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w" podUID="1816f3b7-b324-45cd-add2-9931d7f1eb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 12:38:04.703895 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:04.703420    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:38:04.703895 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:04.703488    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w" podUID="1816f3b7-b324-45cd-add2-9931d7f1eb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 12:38:05.372716 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:05.372680    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:05.706086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:05.703495    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:38:05.706086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:05.703569    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w" podUID="1816f3b7-b324-45cd-add2-9931d7f1eb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 12:38:06.705034 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:06.704502    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:38:06.705034 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:06.704570    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w" podUID="1816f3b7-b324-45cd-add2-9931d7f1eb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 12:38:07.170190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:07.170149    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:07.706535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:07.706495    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 02 12:38:07.706732 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:07.706569    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w" podUID="1816f3b7-b324-45cd-add2-9931d7f1eb26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 02 12:38:08.315073 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:08.315029    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:08.499056 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:08.499021    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:08.823190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:08.823143    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:08.823542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:08.823521    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:08.824369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:08.824350    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:38:09.327348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:09.327315    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:10.155273 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:10.155236    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:11.367127 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.367089    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:11.404778 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.404740    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"]
Nov 02 12:38:11.405328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.405313    5879 cpu_manager.go:414] "containerMap: removing container" podUID="8175b87e-e393-44de-9a73-66b76e696d50" containerName="revoke-keys"
Nov 02 12:38:11.405380 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.405330    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8175b87e-e393-44de-9a73-66b76e696d50" containerName="revoke-keys"
Nov 02 12:38:11.405438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.405432    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="8175b87e-e393-44de-9a73-66b76e696d50" containerName="revoke-keys"
Nov 02 12:38:11.409201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.409179    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:11.432540 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.432496    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"]
Nov 02 12:38:11.443107 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.443077    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:38:11.443107 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.443086    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:38:11.516199 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.516160    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1b05aaf-jf8x9\" (UID: \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:11.516416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.516214    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1b05aaf-jf8x9\" (UID: \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:11.516416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.516343    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm7g6\" (UniqueName: \"kubernetes.io/projected/ef3de9e5-ab1d-464d-811b-3f451ac760c6-kube-api-access-vm7g6\") pod \"maas-api-revoke-sub-d1b05aaf-jf8x9\" (UID: \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:11.617428 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.617319    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm7g6\" (UniqueName: \"kubernetes.io/projected/ef3de9e5-ab1d-464d-811b-3f451ac760c6-kube-api-access-vm7g6\") pod \"maas-api-revoke-sub-d1b05aaf-jf8x9\" (UID: \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:11.617620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.617491    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1b05aaf-jf8x9\" (UID: \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:11.617620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.617545    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1b05aaf-jf8x9\" (UID: \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:11.618520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.618488    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1b05aaf-jf8x9\" (UID: \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:11.620170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.620148    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1b05aaf-jf8x9\" (UID: \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:11.643348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.643315    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm7g6\" (UniqueName: \"kubernetes.io/projected/ef3de9e5-ab1d-464d-811b-3f451ac760c6-kube-api-access-vm7g6\") pod \"maas-api-revoke-sub-d1b05aaf-jf8x9\" (UID: \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:11.723483 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:11.723438    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:12.034234 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:38:12.034196    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3de9e5_ab1d_464d_811b_3f451ac760c6.slice/crio-4f34cb465154705cffa576dd6c46edd837ff99a64a481a08babfae36da812992 WatchSource:0}: Error finding container 4f34cb465154705cffa576dd6c46edd837ff99a64a481a08babfae36da812992: Status 404 returned error can't find the container with id 4f34cb465154705cffa576dd6c46edd837ff99a64a481a08babfae36da812992
Nov 02 12:38:12.034491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:12.034461    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"]
Nov 02 12:38:12.373327 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:12.373288    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9" event={"ID":"ef3de9e5-ab1d-464d-811b-3f451ac760c6","Type":"ContainerStarted","Data":"72b5fc70259abbec7488ed64ddb5cf24154797bc61451e1aa726a27939d8a6fe"}
Nov 02 12:38:12.373730 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:12.373345    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9" event={"ID":"ef3de9e5-ab1d-464d-811b-3f451ac760c6","Type":"ContainerStarted","Data":"4f34cb465154705cffa576dd6c46edd837ff99a64a481a08babfae36da812992"}
Nov 02 12:38:12.418964 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:12.418885    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9" podStartSLOduration=1.418863621 podStartE2EDuration="1.418863621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:12.417481145 +0000 UTC m=+638.524183998" watchObservedRunningTime="2026-11-02 12:38:12.418863621 +0000 UTC m=+638.525566474"
Nov 02 12:38:13.088344 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:13.088257    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:13.383439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:13.383400    5879 generic.go:356] "Generic (PLEG): container finished" podID="ef3de9e5-ab1d-464d-811b-3f451ac760c6" containerID="72b5fc70259abbec7488ed64ddb5cf24154797bc61451e1aa726a27939d8a6fe" exitCode=0
Nov 02 12:38:13.383872 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:13.383468    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9" event={"ID":"ef3de9e5-ab1d-464d-811b-3f451ac760c6","Type":"ContainerDied","Data":"72b5fc70259abbec7488ed64ddb5cf24154797bc61451e1aa726a27939d8a6fe"}
Nov 02 12:38:14.292538 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.292501    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"]
Nov 02 12:38:14.296561 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.296537    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:14.316712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.316683    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"]
Nov 02 12:38:14.451060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.451026    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56bc10a0-64cww\" (UID: \"3a194bf5-efe2-44b4-b546-6251943cc662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:14.451691 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.451134    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56bc10a0-64cww\" (UID: \"3a194bf5-efe2-44b4-b546-6251943cc662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:14.451691 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.451211    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mbz7\" (UniqueName: \"kubernetes.io/projected/3a194bf5-efe2-44b4-b546-6251943cc662-kube-api-access-8mbz7\") pod \"maas-api-revoke-sub-56bc10a0-64cww\" (UID: \"3a194bf5-efe2-44b4-b546-6251943cc662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:14.540787 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.540747    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:14.552239 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.552159    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56bc10a0-64cww\" (UID: \"3a194bf5-efe2-44b4-b546-6251943cc662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:14.552239 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.552216    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mbz7\" (UniqueName: \"kubernetes.io/projected/3a194bf5-efe2-44b4-b546-6251943cc662-kube-api-access-8mbz7\") pod \"maas-api-revoke-sub-56bc10a0-64cww\" (UID: \"3a194bf5-efe2-44b4-b546-6251943cc662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:14.552473 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.552303    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56bc10a0-64cww\" (UID: \"3a194bf5-efe2-44b4-b546-6251943cc662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:14.552900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.552874    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56bc10a0-64cww\" (UID: \"3a194bf5-efe2-44b4-b546-6251943cc662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:14.554739 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.554720    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56bc10a0-64cww\" (UID: \"3a194bf5-efe2-44b4-b546-6251943cc662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:14.578141 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.578112    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mbz7\" (UniqueName: \"kubernetes.io/projected/3a194bf5-efe2-44b4-b546-6251943cc662-kube-api-access-8mbz7\") pod \"maas-api-revoke-sub-56bc10a0-64cww\" (UID: \"3a194bf5-efe2-44b4-b546-6251943cc662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:14.610574 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.610535    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:14.653243 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.653198    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-xks-ca\") pod \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\" (UID: \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\") "
Nov 02 12:38:14.653444 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.653336    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-openshift-ca\") pod \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\" (UID: \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\") "
Nov 02 12:38:14.653444 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.653385    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3de9e5-ab1d-464d-811b-3f451ac760c6-kube-api-access-vm7g6\" (UniqueName: \"kubernetes.io/projected/ef3de9e5-ab1d-464d-811b-3f451ac760c6-kube-api-access-vm7g6\") pod \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\" (UID: \"ef3de9e5-ab1d-464d-811b-3f451ac760c6\") "
Nov 02 12:38:14.653704 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.653680    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-openshift-ca" pod "ef3de9e5-ab1d-464d-811b-3f451ac760c6" (UID: "ef3de9e5-ab1d-464d-811b-3f451ac760c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:14.653844 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.653823    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:14.655593 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.655569    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-xks-ca" pod "ef3de9e5-ab1d-464d-811b-3f451ac760c6" (UID: "ef3de9e5-ab1d-464d-811b-3f451ac760c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:38:14.655685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.655635    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3de9e5-ab1d-464d-811b-3f451ac760c6-kube-api-access-vm7g6" pod "ef3de9e5-ab1d-464d-811b-3f451ac760c6" (UID: "ef3de9e5-ab1d-464d-811b-3f451ac760c6"). InnerVolumeSpecName "kube-api-access-vm7g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:14.754767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.754726    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm7g6\" (UniqueName: \"kubernetes.io/projected/ef3de9e5-ab1d-464d-811b-3f451ac760c6-kube-api-access-vm7g6\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:14.754767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.754759    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3de9e5-ab1d-464d-811b-3f451ac760c6-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:14.825795 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:38:14.825709    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:38:14.893849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.893792    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:14.918155 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:38:14.918124    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a194bf5_efe2_44b4_b546_6251943cc662.slice/crio-8cede3f7277b3170ee0cf696364509f4d0850eec8b11e3191d853fb5483204c7 WatchSource:0}: Error finding container 8cede3f7277b3170ee0cf696364509f4d0850eec8b11e3191d853fb5483204c7: Status 404 returned error can't find the container with id 8cede3f7277b3170ee0cf696364509f4d0850eec8b11e3191d853fb5483204c7
Nov 02 12:38:14.918424 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:14.918397    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"]
Nov 02 12:38:15.337641 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.337606    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"]
Nov 02 12:38:15.338268 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.338226    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ef3de9e5-ab1d-464d-811b-3f451ac760c6" containerName="revoke-keys"
Nov 02 12:38:15.338268 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.338249    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3de9e5-ab1d-464d-811b-3f451ac760c6" containerName="revoke-keys"
Nov 02 12:38:15.338489 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.338386    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3de9e5-ab1d-464d-811b-3f451ac760c6" containerName="revoke-keys"
Nov 02 12:38:15.341843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.341823    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.362808 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.362780    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-95ee26-openshift-default-dockercfg-xjh8d\""
Nov 02 12:38:15.371520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.371449    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"]
Nov 02 12:38:15.399089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.399054    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww" event={"ID":"3a194bf5-efe2-44b4-b546-6251943cc662","Type":"ContainerStarted","Data":"bbd6024cb0a8cfe632a4360cb9b34dfef51ce80b23758ecad17c398662b89acf"}
Nov 02 12:38:15.399267 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.399126    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww" event={"ID":"3a194bf5-efe2-44b4-b546-6251943cc662","Type":"ContainerStarted","Data":"8cede3f7277b3170ee0cf696364509f4d0850eec8b11e3191d853fb5483204c7"}
Nov 02 12:38:15.400847 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.400824    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"
Nov 02 12:38:15.400951 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.400851    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9" event={"ID":"ef3de9e5-ab1d-464d-811b-3f451ac760c6","Type":"ContainerDied","Data":"4f34cb465154705cffa576dd6c46edd837ff99a64a481a08babfae36da812992"}
Nov 02 12:38:15.400951 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.400900    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f34cb465154705cffa576dd6c46edd837ff99a64a481a08babfae36da812992"
Nov 02 12:38:15.443967 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.443908    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww" podStartSLOduration=1.443892669 podStartE2EDuration="1.443892669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:15.44250276 +0000 UTC m=+641.549205612" watchObservedRunningTime="2026-11-02 12:38:15.443892669 +0000 UTC m=+641.550595520"
Nov 02 12:38:15.464123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.464084    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.464123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.464120    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq52j\" (UniqueName: \"kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-kube-api-access-nq52j\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.464739 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.464176    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.464739 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.464374    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.464739 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.464415    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.464739 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.464541    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.464739 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.464641    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e00db56-210d-4523-bc46-779067bac1b5-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.464739 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.464673    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.464739 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.464700    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e00db56-210d-4523-bc46-779067bac1b5-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.565557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.565517    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.565768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.565582    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.565768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.565605    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.565768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.565647    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.565768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.565703    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e00db56-210d-4523-bc46-779067bac1b5-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.565768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.565726    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.565768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.565749    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e00db56-210d-4523-bc46-779067bac1b5-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.566122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.566039    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.566177 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.566122    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.566177 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.566153    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.566249 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.566207    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.566249 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.565801    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.566349 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.566281    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq52j\" (UniqueName: \"kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-kube-api-access-nq52j\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.566461 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.566439    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e00db56-210d-4523-bc46-779067bac1b5-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.568649 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.568619    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.568779 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.568750    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e00db56-210d-4523-bc46-779067bac1b5-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.599503 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.599431    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.601783 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.601754    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq52j\" (UniqueName: \"kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-kube-api-access-nq52j\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.655473 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.655434    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:15.975507 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:38:15.975465    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e00db56_210d_4523_bc46_779067bac1b5.slice/crio-594d74a5172abff44be200069877ecd7638f15b4a8be8d4d222714504fd88c41 WatchSource:0}: Error finding container 594d74a5172abff44be200069877ecd7638f15b4a8be8d4d222714504fd88c41: Status 404 returned error can't find the container with id 594d74a5172abff44be200069877ecd7638f15b4a8be8d4d222714504fd88c41
Nov 02 12:38:15.975825 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.975799    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"]
Nov 02 12:38:15.978797 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.978757    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:15.978898 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.978848    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:15.978898 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:15.978889    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:16.413103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:16.413040    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67" event={"ID":"3e00db56-210d-4523-bc46-779067bac1b5","Type":"ContainerStarted","Data":"5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475"}
Nov 02 12:38:16.413915 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:16.413860    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67" event={"ID":"3e00db56-210d-4523-bc46-779067bac1b5","Type":"ContainerStarted","Data":"594d74a5172abff44be200069877ecd7638f15b4a8be8d4d222714504fd88c41"}
Nov 02 12:38:16.415417 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:16.415229    5879 generic.go:356] "Generic (PLEG): container finished" podID="3a194bf5-efe2-44b4-b546-6251943cc662" containerID="bbd6024cb0a8cfe632a4360cb9b34dfef51ce80b23758ecad17c398662b89acf" exitCode=0
Nov 02 12:38:16.415417 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:16.415319    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww" event={"ID":"3a194bf5-efe2-44b4-b546-6251943cc662","Type":"ContainerDied","Data":"bbd6024cb0a8cfe632a4360cb9b34dfef51ce80b23758ecad17c398662b89acf"}
Nov 02 12:38:16.499024 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:16.494624    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67" podStartSLOduration=1.494603289 podStartE2EDuration="1.494603289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:16.467414738 +0000 UTC m=+642.574117591" watchObservedRunningTime="2026-11-02 12:38:16.494603289 +0000 UTC m=+642.601306143"
Nov 02 12:38:16.656476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:16.656393    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:16.795683 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:16.795586    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:17.671027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:17.657675    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 12:38:17.671027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:17.657759    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67" podUID="3e00db56-210d-4523-bc46-779067bac1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 12:38:17.780052 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:17.776566    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:17.932020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:17.931625    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-xks-ca\") pod \"3a194bf5-efe2-44b4-b546-6251943cc662\" (UID: \"3a194bf5-efe2-44b4-b546-6251943cc662\") "
Nov 02 12:38:17.932020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:17.931744    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a194bf5-efe2-44b4-b546-6251943cc662-kube-api-access-8mbz7\" (UniqueName: \"kubernetes.io/projected/3a194bf5-efe2-44b4-b546-6251943cc662-kube-api-access-8mbz7\") pod \"3a194bf5-efe2-44b4-b546-6251943cc662\" (UID: \"3a194bf5-efe2-44b4-b546-6251943cc662\") "
Nov 02 12:38:17.932020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:17.931790    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-openshift-ca\") pod \"3a194bf5-efe2-44b4-b546-6251943cc662\" (UID: \"3a194bf5-efe2-44b4-b546-6251943cc662\") "
Nov 02 12:38:17.974022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:17.965842    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-openshift-ca" pod "3a194bf5-efe2-44b4-b546-6251943cc662" (UID: "3a194bf5-efe2-44b4-b546-6251943cc662"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:17.974022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:17.966716    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:17.974959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:17.974329    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-xks-ca" pod "3a194bf5-efe2-44b4-b546-6251943cc662" (UID: "3a194bf5-efe2-44b4-b546-6251943cc662"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:38:17.980286 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:17.980213    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a194bf5-efe2-44b4-b546-6251943cc662-kube-api-access-8mbz7" pod "3a194bf5-efe2-44b4-b546-6251943cc662" (UID: "3a194bf5-efe2-44b4-b546-6251943cc662"). InnerVolumeSpecName "kube-api-access-8mbz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:18.068275 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:18.068229    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a194bf5-efe2-44b4-b546-6251943cc662-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:18.068430 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:18.068277    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mbz7\" (UniqueName: \"kubernetes.io/projected/3a194bf5-efe2-44b4-b546-6251943cc662-kube-api-access-8mbz7\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:18.456908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:18.456095    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"
Nov 02 12:38:18.456908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:18.456114    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww" event={"ID":"3a194bf5-efe2-44b4-b546-6251943cc662","Type":"ContainerDied","Data":"8cede3f7277b3170ee0cf696364509f4d0850eec8b11e3191d853fb5483204c7"}
Nov 02 12:38:18.456908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:18.456263    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cede3f7277b3170ee0cf696364509f4d0850eec8b11e3191d853fb5483204c7"
Nov 02 12:38:18.656981 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:18.656419    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 12:38:18.656981 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:18.656491    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67" podUID="3e00db56-210d-4523-bc46-779067bac1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 12:38:18.721075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:18.720781    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:19.656222 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:19.656178    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 12:38:19.656421 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:19.656258    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67" podUID="3e00db56-210d-4523-bc46-779067bac1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 12:38:20.479159 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:20.479123    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:20.655981 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:20.655946    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 12:38:20.656197 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:20.656034    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67" podUID="3e00db56-210d-4523-bc46-779067bac1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 12:38:21.656254 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:21.656222    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 02 12:38:21.656637 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:21.656275    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67" podUID="3e00db56-210d-4523-bc46-779067bac1b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 02 12:38:22.183398 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:22.183364    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:22.816120 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:22.816077    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:22.817458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:22.816780    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:22.818267 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:22.818241    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 12:38:23.827694 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:23.827645    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:24.629589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.629549    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"]
Nov 02 12:38:24.630327 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.630299    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3a194bf5-efe2-44b4-b546-6251943cc662" containerName="revoke-keys"
Nov 02 12:38:24.630327 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.630330    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a194bf5-efe2-44b4-b546-6251943cc662" containerName="revoke-keys"
Nov 02 12:38:24.630460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.630449    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a194bf5-efe2-44b4-b546-6251943cc662" containerName="revoke-keys"
Nov 02 12:38:24.634271 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.634234    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:24.658884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.658850    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:38:24.670334 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.670297    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:38:24.673889 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.673855    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"]
Nov 02 12:38:24.768393 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.768348    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8395a3b-ndnlv\" (UID: \"961ca2b3-89b7-474f-a146-bab598bc9d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:24.768599 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.768488    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xpxn\" (UniqueName: \"kubernetes.io/projected/961ca2b3-89b7-474f-a146-bab598bc9d39-kube-api-access-4xpxn\") pod \"maas-api-revoke-sub-a8395a3b-ndnlv\" (UID: \"961ca2b3-89b7-474f-a146-bab598bc9d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:24.768674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.768652    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8395a3b-ndnlv\" (UID: \"961ca2b3-89b7-474f-a146-bab598bc9d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:24.869699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.869659    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpxn\" (UniqueName: \"kubernetes.io/projected/961ca2b3-89b7-474f-a146-bab598bc9d39-kube-api-access-4xpxn\") pod \"maas-api-revoke-sub-a8395a3b-ndnlv\" (UID: \"961ca2b3-89b7-474f-a146-bab598bc9d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:24.870141 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.869766    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8395a3b-ndnlv\" (UID: \"961ca2b3-89b7-474f-a146-bab598bc9d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:24.870141 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.869814    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8395a3b-ndnlv\" (UID: \"961ca2b3-89b7-474f-a146-bab598bc9d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:24.870637 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.870612    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8395a3b-ndnlv\" (UID: \"961ca2b3-89b7-474f-a146-bab598bc9d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:24.872331 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.872315    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8395a3b-ndnlv\" (UID: \"961ca2b3-89b7-474f-a146-bab598bc9d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:24.895166 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.895096    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpxn\" (UniqueName: \"kubernetes.io/projected/961ca2b3-89b7-474f-a146-bab598bc9d39-kube-api-access-4xpxn\") pod \"maas-api-revoke-sub-a8395a3b-ndnlv\" (UID: \"961ca2b3-89b7-474f-a146-bab598bc9d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:24.947726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:24.947687    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:25.253968 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:25.253937    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"]
Nov 02 12:38:25.254122 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:38:25.254028    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961ca2b3_89b7_474f_a146_bab598bc9d39.slice/crio-191a93253583fcef18a5c65cc976f604cd3581029b9a4edcaecb2bf243ccb846 WatchSource:0}: Error finding container 191a93253583fcef18a5c65cc976f604cd3581029b9a4edcaecb2bf243ccb846: Status 404 returned error can't find the container with id 191a93253583fcef18a5c65cc976f604cd3581029b9a4edcaecb2bf243ccb846
Nov 02 12:38:25.460413 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:25.460371    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:25.502382 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:25.502301    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv" event={"ID":"961ca2b3-89b7-474f-a146-bab598bc9d39","Type":"ContainerStarted","Data":"191a93253583fcef18a5c65cc976f604cd3581029b9a4edcaecb2bf243ccb846"}
Nov 02 12:38:25.720409 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:25.720372    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:26.511269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:26.511167    5879 generic.go:356] "Generic (PLEG): container finished" podID="961ca2b3-89b7-474f-a146-bab598bc9d39" containerID="534b8b6c75e5777bf0564f01761b8ecc87b88fd364018ae67db0ce11b558fc6a" exitCode=0
Nov 02 12:38:26.511647 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:26.511260    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv" event={"ID":"961ca2b3-89b7-474f-a146-bab598bc9d39","Type":"ContainerDied","Data":"534b8b6c75e5777bf0564f01761b8ecc87b88fd364018ae67db0ce11b558fc6a"}
Nov 02 12:38:27.238711 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.238670    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:27.502072 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.501926    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"]
Nov 02 12:38:27.506100 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.506075    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.529131 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.528862    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-068756-openshift-default-dockercfg-kp7kc\""
Nov 02 12:38:27.547360 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.547322    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"]
Nov 02 12:38:27.606234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.606196    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.606425 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.606267    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.606425 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.606331    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.606425 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.606368    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270ac0a3-1976-47e7-8879-22a2ae545377-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.606587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.606455    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d47c9\" (UniqueName: \"kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-kube-api-access-d47c9\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.606587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.606495    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.606587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.606521    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270ac0a3-1976-47e7-8879-22a2ae545377-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.606587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.606582    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.606775 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.606617    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.708386 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.708352    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.708606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.708397    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.708606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.708435    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.708606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.708462    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270ac0a3-1976-47e7-8879-22a2ae545377-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.708606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.708546    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d47c9\" (UniqueName: \"kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-kube-api-access-d47c9\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.708606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.708581    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.708606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.708603    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270ac0a3-1976-47e7-8879-22a2ae545377-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.708963 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.708659    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.708963 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.708691    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.708963 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.708914    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.709180 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.709066    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.709554 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.709499    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.709554 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.709505    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.709729 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.709654    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270ac0a3-1976-47e7-8879-22a2ae545377-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.711704 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.711677    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.712132 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.712111    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270ac0a3-1976-47e7-8879-22a2ae545377-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.734365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.734321    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d47c9\" (UniqueName: \"kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-kube-api-access-d47c9\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.734505 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.734452    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.737029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.737007    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:27.810784 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.809632    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961ca2b3-89b7-474f-a146-bab598bc9d39-kube-api-access-4xpxn\" (UniqueName: \"kubernetes.io/projected/961ca2b3-89b7-474f-a146-bab598bc9d39-kube-api-access-4xpxn\") pod \"961ca2b3-89b7-474f-a146-bab598bc9d39\" (UID: \"961ca2b3-89b7-474f-a146-bab598bc9d39\") "
Nov 02 12:38:27.810784 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.809727    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-openshift-ca\") pod \"961ca2b3-89b7-474f-a146-bab598bc9d39\" (UID: \"961ca2b3-89b7-474f-a146-bab598bc9d39\") "
Nov 02 12:38:27.810784 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.809872    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-xks-ca\") pod \"961ca2b3-89b7-474f-a146-bab598bc9d39\" (UID: \"961ca2b3-89b7-474f-a146-bab598bc9d39\") "
Nov 02 12:38:27.812665 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.812384    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-openshift-ca" pod "961ca2b3-89b7-474f-a146-bab598bc9d39" (UID: "961ca2b3-89b7-474f-a146-bab598bc9d39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:27.822601 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.822539    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-xks-ca" pod "961ca2b3-89b7-474f-a146-bab598bc9d39" (UID: "961ca2b3-89b7-474f-a146-bab598bc9d39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:38:27.828448 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.828409    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961ca2b3-89b7-474f-a146-bab598bc9d39-kube-api-access-4xpxn" pod "961ca2b3-89b7-474f-a146-bab598bc9d39" (UID: "961ca2b3-89b7-474f-a146-bab598bc9d39"). InnerVolumeSpecName "kube-api-access-4xpxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:27.852653 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.850733    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:27.911499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.911454    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:27.911499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.911498    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xpxn\" (UniqueName: \"kubernetes.io/projected/961ca2b3-89b7-474f-a146-bab598bc9d39-kube-api-access-4xpxn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:27.911718 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:27.911515    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/961ca2b3-89b7-474f-a146-bab598bc9d39-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:28.205958 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:38:28.205909    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270ac0a3_1976_47e7_8879_22a2ae545377.slice/crio-8d83aeb20e98d53361f8dba771bce3b6eb1557bd289d126dcabc8e809794900f WatchSource:0}: Error finding container 8d83aeb20e98d53361f8dba771bce3b6eb1557bd289d126dcabc8e809794900f: Status 404 returned error can't find the container with id 8d83aeb20e98d53361f8dba771bce3b6eb1557bd289d126dcabc8e809794900f
Nov 02 12:38:28.221263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.221168    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"]
Nov 02 12:38:28.222948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.222898    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:28.223167 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.222984    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:28.223167 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.223049    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:28.550218 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.550106    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv" event={"ID":"961ca2b3-89b7-474f-a146-bab598bc9d39","Type":"ContainerDied","Data":"191a93253583fcef18a5c65cc976f604cd3581029b9a4edcaecb2bf243ccb846"}
Nov 02 12:38:28.550849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.550827    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="191a93253583fcef18a5c65cc976f604cd3581029b9a4edcaecb2bf243ccb846"
Nov 02 12:38:28.552880 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.551118    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"
Nov 02 12:38:28.555670 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.555639    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql" event={"ID":"270ac0a3-1976-47e7-8879-22a2ae545377","Type":"ContainerStarted","Data":"6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca"}
Nov 02 12:38:28.555835 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.555695    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql" event={"ID":"270ac0a3-1976-47e7-8879-22a2ae545377","Type":"ContainerStarted","Data":"8d83aeb20e98d53361f8dba771bce3b6eb1557bd289d126dcabc8e809794900f"}
Nov 02 12:38:28.631021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.630133    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql" podStartSLOduration=1.630112842 podStartE2EDuration="1.630112842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:28.628852709 +0000 UTC m=+654.735555567" watchObservedRunningTime="2026-11-02 12:38:28.630112842 +0000 UTC m=+654.736815694"
Nov 02 12:38:28.738020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.737486    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:28.852250 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.852148    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:28.987019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:28.986149    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:29.186027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:29.179798    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:29.375652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:29.375608    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:29.853425 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:29.853395    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:29.853820 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:29.853445    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql" podUID="270ac0a3-1976-47e7-8879-22a2ae545377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 12:38:30.851856 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:30.851822    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:30.852053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:30.851882    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql" podUID="270ac0a3-1976-47e7-8879-22a2ae545377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 12:38:31.306028 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:31.301465    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:31.851734 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:31.851700    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:31.851927 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:31.851761    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql" podUID="270ac0a3-1976-47e7-8879-22a2ae545377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 12:38:32.852106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:32.852068    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:32.852633 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:32.852129    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql" podUID="270ac0a3-1976-47e7-8879-22a2ae545377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 12:38:33.052351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:33.052302    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:33.507597 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:33.507546    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:33.852161 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:33.852072    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:33.852161 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:33.852140    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql" podUID="270ac0a3-1976-47e7-8879-22a2ae545377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 12:38:33.923869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:33.923835    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:34.820982 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:34.820945    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:34.852681 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:34.852645    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 02 12:38:34.853270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:34.853230    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql" podUID="270ac0a3-1976-47e7-8879-22a2ae545377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 02 12:38:35.857074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:35.857042    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:35.857554 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:35.857495    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:35.858414 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:35.858394    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:38:36.545229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.545190    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"]
Nov 02 12:38:36.546035 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.546009    5879 cpu_manager.go:414] "containerMap: removing container" podUID="961ca2b3-89b7-474f-a146-bab598bc9d39" containerName="revoke-keys"
Nov 02 12:38:36.546035 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.546034    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961ca2b3-89b7-474f-a146-bab598bc9d39" containerName="revoke-keys"
Nov 02 12:38:36.546272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.546193    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="961ca2b3-89b7-474f-a146-bab598bc9d39" containerName="revoke-keys"
Nov 02 12:38:36.546448 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.546423    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:36.551557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.551530    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.578110 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.578072    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.578302 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.578140    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4mwc\" (UniqueName: \"kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-kube-api-access-h4mwc\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.578302 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.578163    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.578302 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.578195    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b970dd-6fcd-4e7a-913a-03abc88d0969-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.578468 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.578329    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.578468 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.578454    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.578553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.578485    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.578553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.578513    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.578644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.578568    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.581440 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.581410    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"]
Nov 02 12:38:36.679362 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.679327    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4mwc\" (UniqueName: \"kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-kube-api-access-h4mwc\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.679565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.679375    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.679565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.679411    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b970dd-6fcd-4e7a-913a-03abc88d0969-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.679565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.679462    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.679565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.679549    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.679804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.679579    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.679804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.679608    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.679804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.679676    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.679961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.679802    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.679961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.679857    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.679961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.679956    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.680221 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.680197    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.680379 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.680356    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.680619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.680555    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b970dd-6fcd-4e7a-913a-03abc88d0969-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.682185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.682162    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.682378 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.682361    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.706530 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.706490    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4mwc\" (UniqueName: \"kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-kube-api-access-h4mwc\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.709215 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.709191    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-fx2tt\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:36.882277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:36.882243    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:37.203277 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:38:37.203249    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b970dd_6fcd_4e7a_913a_03abc88d0969.slice/crio-4a58064168c841f7524622ee25d35eaa49b1d3c16e92ddd541e5794e345fe4c1 WatchSource:0}: Error finding container 4a58064168c841f7524622ee25d35eaa49b1d3c16e92ddd541e5794e345fe4c1: Status 404 returned error can't find the container with id 4a58064168c841f7524622ee25d35eaa49b1d3c16e92ddd541e5794e345fe4c1
Nov 02 12:38:37.203860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:37.203833    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"]
Nov 02 12:38:37.205973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:37.205924    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:37.206102 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:37.206040    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:37.206102 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:37.206087    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:37.650366 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:37.650322    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" event={"ID":"50b970dd-6fcd-4e7a-913a-03abc88d0969","Type":"ContainerStarted","Data":"09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc"}
Nov 02 12:38:37.650587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:37.650399    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" event={"ID":"50b970dd-6fcd-4e7a-913a-03abc88d0969","Type":"ContainerStarted","Data":"4a58064168c841f7524622ee25d35eaa49b1d3c16e92ddd541e5794e345fe4c1"}
Nov 02 12:38:37.701500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:37.701438    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" podStartSLOduration=1.701420122 podStartE2EDuration="1.701420122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:37.701098302 +0000 UTC m=+663.807801158" watchObservedRunningTime="2026-11-02 12:38:37.701420122 +0000 UTC m=+663.808122974"
Nov 02 12:38:37.883452 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:37.883332    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:38.673362 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:38.673302    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:38.903946 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:38.903892    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fx2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:38.904577 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:38.904523    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 12:38:39.885500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:39.885462    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fx2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:39.885825 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:39.885786    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 12:38:40.734269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:40.734237    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:40.885016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:40.883030    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fx2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:40.885016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:40.883097    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 12:38:41.897580 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:41.897540    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fx2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:41.898212 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:41.898175    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 12:38:42.534883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:42.534849    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:42.901495 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:42.901454    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fx2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:42.902545 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:42.902102    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 12:38:43.883291 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:43.883255    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fx2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:43.883531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:43.883312    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 12:38:44.379312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:44.379277    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:44.825646 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:38:44.825550    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:38:44.883320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:44.882826    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fx2tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 12:38:44.883320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:44.882889    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 12:38:45.875797 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:45.875756    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:46.025383 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.025332    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:46.556680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.556635    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"]
Nov 02 12:38:46.561482 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.561442    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.657428 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.657382    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"]
Nov 02 12:38:46.673392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.673349    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.673392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.673387    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.673659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.673409    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.673659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.673537    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.673659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.673618    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.673841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.673708    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344cbaff-2812-4910-a6dc-7c33fd030d39-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.673841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.673762    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.673841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.673836    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsc4m\" (UniqueName: \"kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-kube-api-access-gsc4m\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.674023 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.673885    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.752077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.751530    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:46.752419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.752392    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:46.753663 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.753640    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:38:46.774945 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.774901    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.774945 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.774938    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344cbaff-2812-4910-a6dc-7c33fd030d39-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.775211 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.774971    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.775211 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.775021    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsc4m\" (UniqueName: \"kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-kube-api-access-gsc4m\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.775211 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.775095    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.775211 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.775149    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.775412 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.775331    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.775412 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.775373    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.775412 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.775376    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.775546 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.775501    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.775546 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.775519    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.775654 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.775630    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.776046 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.776023    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.776349 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.776328    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344cbaff-2812-4910-a6dc-7c33fd030d39-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.778084 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.778040    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.778209 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.778189    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.827730 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.827596    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.828569 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.828518    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsc4m\" (UniqueName: \"kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-kube-api-access-gsc4m\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:46.879392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:46.879362    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:47.372073 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:47.371949    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"]
Nov 02 12:38:47.381568 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:38:47.381530    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod344cbaff_2812_4910_a6dc_7c33fd030d39.slice/crio-0cd74b774118ccf4810cb042d5b3ebfa0130fbfc61ce0c2ca6c1507e2c2c2cf1 WatchSource:0}: Error finding container 0cd74b774118ccf4810cb042d5b3ebfa0130fbfc61ce0c2ca6c1507e2c2c2cf1: Status 404 returned error can't find the container with id 0cd74b774118ccf4810cb042d5b3ebfa0130fbfc61ce0c2ca6c1507e2c2c2cf1
Nov 02 12:38:47.390245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:47.390191    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:47.390523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:47.390494    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:47.390728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:47.390692    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:38:47.803077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:47.800901    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4" event={"ID":"344cbaff-2812-4910-a6dc-7c33fd030d39","Type":"ContainerStarted","Data":"31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c"}
Nov 02 12:38:47.803077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:47.800958    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4" event={"ID":"344cbaff-2812-4910-a6dc-7c33fd030d39","Type":"ContainerStarted","Data":"0cd74b774118ccf4810cb042d5b3ebfa0130fbfc61ce0c2ca6c1507e2c2c2cf1"}
Nov 02 12:38:47.856612 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:47.856547    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4" podStartSLOduration=1.8565251809999999 podStartE2EDuration="1.856525181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:38:47.856356172 +0000 UTC m=+673.963059038" watchObservedRunningTime="2026-11-02 12:38:47.856525181 +0000 UTC m=+673.963228034"
Nov 02 12:38:47.881744 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:47.881694    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:48.082352 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:48.082205    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:48.902017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:48.894371    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 12:38:48.902017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:48.894436    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4" podUID="344cbaff-2812-4910-a6dc-7c33fd030d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 12:38:49.875366 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:49.875329    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:49.880511 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:49.880477    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 12:38:49.880649 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:49.880529    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4" podUID="344cbaff-2812-4910-a6dc-7c33fd030d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 12:38:50.880240 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:50.880206    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 12:38:50.880636 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:50.880260    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4" podUID="344cbaff-2812-4910-a6dc-7c33fd030d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 12:38:51.656149 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:51.656120    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:51.880304 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:51.880274    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 12:38:51.880677 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:51.880328    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4" podUID="344cbaff-2812-4910-a6dc-7c33fd030d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 12:38:52.116655 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.116619    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"]
Nov 02 12:38:52.121129 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.121110    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:52.140125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.140094    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"]
Nov 02 12:38:52.150117 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.150083    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:38:52.150117 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.150118    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:38:52.180699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.180668    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhc99\" (UniqueName: \"kubernetes.io/projected/b4c9c358-856a-4200-8cc3-80a379e3fe14-kube-api-access-fhc99\") pod \"maas-api-revoke-sub-6f7985cb-55qfg\" (UID: \"b4c9c358-856a-4200-8cc3-80a379e3fe14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:52.180871 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.180832    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f7985cb-55qfg\" (UID: \"b4c9c358-856a-4200-8cc3-80a379e3fe14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:52.180969 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.180949    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f7985cb-55qfg\" (UID: \"b4c9c358-856a-4200-8cc3-80a379e3fe14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:52.282675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.282636    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f7985cb-55qfg\" (UID: \"b4c9c358-856a-4200-8cc3-80a379e3fe14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:52.284295 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.284268    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhc99\" (UniqueName: \"kubernetes.io/projected/b4c9c358-856a-4200-8cc3-80a379e3fe14-kube-api-access-fhc99\") pod \"maas-api-revoke-sub-6f7985cb-55qfg\" (UID: \"b4c9c358-856a-4200-8cc3-80a379e3fe14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:52.284890 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.284868    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f7985cb-55qfg\" (UID: \"b4c9c358-856a-4200-8cc3-80a379e3fe14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:52.304171 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.285729    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f7985cb-55qfg\" (UID: \"b4c9c358-856a-4200-8cc3-80a379e3fe14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:52.307756 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.307641    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f7985cb-55qfg\" (UID: \"b4c9c358-856a-4200-8cc3-80a379e3fe14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:52.310290 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.310256    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhc99\" (UniqueName: \"kubernetes.io/projected/b4c9c358-856a-4200-8cc3-80a379e3fe14-kube-api-access-fhc99\") pod \"maas-api-revoke-sub-6f7985cb-55qfg\" (UID: \"b4c9c358-856a-4200-8cc3-80a379e3fe14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:52.435897 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.435406    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:52.807578 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.807213    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"]
Nov 02 12:38:52.808886 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:38:52.808855    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c9c358_856a_4200_8cc3_80a379e3fe14.slice/crio-20715024e576cf0a9ee1a8f15bcae08b5636abd955c811468e885da6a8f990a5 WatchSource:0}: Error finding container 20715024e576cf0a9ee1a8f15bcae08b5636abd955c811468e885da6a8f990a5: Status 404 returned error can't find the container with id 20715024e576cf0a9ee1a8f15bcae08b5636abd955c811468e885da6a8f990a5
Nov 02 12:38:52.853292 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.853252    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg" event={"ID":"b4c9c358-856a-4200-8cc3-80a379e3fe14","Type":"ContainerStarted","Data":"20715024e576cf0a9ee1a8f15bcae08b5636abd955c811468e885da6a8f990a5"}
Nov 02 12:38:52.880091 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.880059    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 12:38:52.880259 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:52.880110    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4" podUID="344cbaff-2812-4910-a6dc-7c33fd030d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 12:38:53.611075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:53.610871    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:53.861759 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:53.861693    5879 generic.go:356] "Generic (PLEG): container finished" podID="b4c9c358-856a-4200-8cc3-80a379e3fe14" containerID="8deb9f9fafd244d6617433b055097223ac8b9a9997cfa0186017cb5f469456ed" exitCode=0
Nov 02 12:38:53.861894 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:53.861758    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg" event={"ID":"b4c9c358-856a-4200-8cc3-80a379e3fe14","Type":"ContainerDied","Data":"8deb9f9fafd244d6617433b055097223ac8b9a9997cfa0186017cb5f469456ed"}
Nov 02 12:38:53.879644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:53.879620    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 12:38:53.879782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:53.879668    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4" podUID="344cbaff-2812-4910-a6dc-7c33fd030d39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 12:38:54.213109 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:54.212883    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:54.885606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:54.885579    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:54.886060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:54.886009    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:54.886927 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:54.886908    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:38:55.031550 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.031520    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:55.141308 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.141211    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4c9c358-856a-4200-8cc3-80a379e3fe14-kube-api-access-fhc99\" (UniqueName: \"kubernetes.io/projected/b4c9c358-856a-4200-8cc3-80a379e3fe14-kube-api-access-fhc99\") pod \"b4c9c358-856a-4200-8cc3-80a379e3fe14\" (UID: \"b4c9c358-856a-4200-8cc3-80a379e3fe14\") "
Nov 02 12:38:55.141462 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.141316    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-xks-ca\") pod \"b4c9c358-856a-4200-8cc3-80a379e3fe14\" (UID: \"b4c9c358-856a-4200-8cc3-80a379e3fe14\") "
Nov 02 12:38:55.141462 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.141363    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-openshift-ca\") pod \"b4c9c358-856a-4200-8cc3-80a379e3fe14\" (UID: \"b4c9c358-856a-4200-8cc3-80a379e3fe14\") "
Nov 02 12:38:55.141763 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.141736    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-openshift-ca" pod "b4c9c358-856a-4200-8cc3-80a379e3fe14" (UID: "b4c9c358-856a-4200-8cc3-80a379e3fe14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:38:55.141896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.141866    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:55.143506 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.143485    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4c9c358-856a-4200-8cc3-80a379e3fe14-kube-api-access-fhc99" pod "b4c9c358-856a-4200-8cc3-80a379e3fe14" (UID: "b4c9c358-856a-4200-8cc3-80a379e3fe14"). InnerVolumeSpecName "kube-api-access-fhc99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:38:55.143624 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.143602    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-xks-ca" pod "b4c9c358-856a-4200-8cc3-80a379e3fe14" (UID: "b4c9c358-856a-4200-8cc3-80a379e3fe14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:38:55.242879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.242844    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhc99\" (UniqueName: \"kubernetes.io/projected/b4c9c358-856a-4200-8cc3-80a379e3fe14-kube-api-access-fhc99\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:55.242879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.242873    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4c9c358-856a-4200-8cc3-80a379e3fe14-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:38:55.615719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.615689    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:55.874582 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.874494    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"
Nov 02 12:38:55.874582 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.874508    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg" event={"ID":"b4c9c358-856a-4200-8cc3-80a379e3fe14","Type":"ContainerDied","Data":"20715024e576cf0a9ee1a8f15bcae08b5636abd955c811468e885da6a8f990a5"}
Nov 02 12:38:55.874582 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:55.874565    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20715024e576cf0a9ee1a8f15bcae08b5636abd955c811468e885da6a8f990a5"
Nov 02 12:38:57.312777 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:57.312742    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:38:59.059750 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:38:59.059717    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:00.749736 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:00.749706    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:01.214453 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:01.214414    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:02.535339 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:02.535291    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:03.615346 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:03.615307    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:03.733719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:03.733671    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:04.277567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.277528    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:04.793216 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.793182    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"]
Nov 02 12:39:04.793728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.793705    5879 cpu_manager.go:414] "containerMap: removing container" podUID="b4c9c358-856a-4200-8cc3-80a379e3fe14" containerName="revoke-keys"
Nov 02 12:39:04.793728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.793718    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4c9c358-856a-4200-8cc3-80a379e3fe14" containerName="revoke-keys"
Nov 02 12:39:04.793836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.793824    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4c9c358-856a-4200-8cc3-80a379e3fe14" containerName="revoke-keys"
Nov 02 12:39:04.799310 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.799279    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:04.818483 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.818434    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"]
Nov 02 12:39:04.828866 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.828834    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:39:04.829110 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.829009    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:39:04.863590 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.863534    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ba245f-gfb5q\" (UID: \"e1dcd822-a886-418d-8f63-6b7daa823f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:04.863933 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.863908    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ba245f-gfb5q\" (UID: \"e1dcd822-a886-418d-8f63-6b7daa823f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:04.864237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.864218    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcdtx\" (UniqueName: \"kubernetes.io/projected/e1dcd822-a886-418d-8f63-6b7daa823f92-kube-api-access-gcdtx\") pod \"maas-api-revoke-sub-30ba245f-gfb5q\" (UID: \"e1dcd822-a886-418d-8f63-6b7daa823f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:04.966394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.966354    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ba245f-gfb5q\" (UID: \"e1dcd822-a886-418d-8f63-6b7daa823f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:04.966611 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.966415    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ba245f-gfb5q\" (UID: \"e1dcd822-a886-418d-8f63-6b7daa823f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:04.966611 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.966471    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcdtx\" (UniqueName: \"kubernetes.io/projected/e1dcd822-a886-418d-8f63-6b7daa823f92-kube-api-access-gcdtx\") pod \"maas-api-revoke-sub-30ba245f-gfb5q\" (UID: \"e1dcd822-a886-418d-8f63-6b7daa823f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:04.967674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.967629    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ba245f-gfb5q\" (UID: \"e1dcd822-a886-418d-8f63-6b7daa823f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:04.969790 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.969759    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ba245f-gfb5q\" (UID: \"e1dcd822-a886-418d-8f63-6b7daa823f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:04.992569 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:04.992495    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcdtx\" (UniqueName: \"kubernetes.io/projected/e1dcd822-a886-418d-8f63-6b7daa823f92-kube-api-access-gcdtx\") pod \"maas-api-revoke-sub-30ba245f-gfb5q\" (UID: \"e1dcd822-a886-418d-8f63-6b7daa823f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:05.117671 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:05.117640    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:05.474441 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:05.474414    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"]
Nov 02 12:39:05.474553 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:39:05.474485    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1dcd822_a886_418d_8f63_6b7daa823f92.slice/crio-40d53946ff5197f15d4bd7c4516d5cde28da0b30e8eaa9c0c853878594c26c38 WatchSource:0}: Error finding container 40d53946ff5197f15d4bd7c4516d5cde28da0b30e8eaa9c0c853878594c26c38: Status 404 returned error can't find the container with id 40d53946ff5197f15d4bd7c4516d5cde28da0b30e8eaa9c0c853878594c26c38
Nov 02 12:39:05.944915 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:05.944873    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q" event={"ID":"e1dcd822-a886-418d-8f63-6b7daa823f92","Type":"ContainerStarted","Data":"5552696d7f9c546810569a735b490b40e809ea207e5dcf8722ae0ce6dfa1278a"}
Nov 02 12:39:05.945339 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:05.944928    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q" event={"ID":"e1dcd822-a886-418d-8f63-6b7daa823f92","Type":"ContainerStarted","Data":"40d53946ff5197f15d4bd7c4516d5cde28da0b30e8eaa9c0c853878594c26c38"}
Nov 02 12:39:05.990360 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:05.990308    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q" podStartSLOduration=1.9902941360000002 podStartE2EDuration="1.990294136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:05.987934131 +0000 UTC m=+692.094636983" watchObservedRunningTime="2026-11-02 12:39:05.990294136 +0000 UTC m=+692.096996987"
Nov 02 12:39:06.143959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:06.143867    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:06.952417 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:06.952383    5879 generic.go:356] "Generic (PLEG): container finished" podID="e1dcd822-a886-418d-8f63-6b7daa823f92" containerID="5552696d7f9c546810569a735b490b40e809ea207e5dcf8722ae0ce6dfa1278a" exitCode=0
Nov 02 12:39:06.952876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:06.952468    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q" event={"ID":"e1dcd822-a886-418d-8f63-6b7daa823f92","Type":"ContainerDied","Data":"5552696d7f9c546810569a735b490b40e809ea207e5dcf8722ae0ce6dfa1278a"}
Nov 02 12:39:07.765939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:07.765907    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:08.106755 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.106726    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:08.205791 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.205751    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-xks-ca\") pod \"e1dcd822-a886-418d-8f63-6b7daa823f92\" (UID: \"e1dcd822-a886-418d-8f63-6b7daa823f92\") "
Nov 02 12:39:08.206041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.205924    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1dcd822-a886-418d-8f63-6b7daa823f92-kube-api-access-gcdtx\" (UniqueName: \"kubernetes.io/projected/e1dcd822-a886-418d-8f63-6b7daa823f92-kube-api-access-gcdtx\") pod \"e1dcd822-a886-418d-8f63-6b7daa823f92\" (UID: \"e1dcd822-a886-418d-8f63-6b7daa823f92\") "
Nov 02 12:39:08.206041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.205950    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-openshift-ca\") pod \"e1dcd822-a886-418d-8f63-6b7daa823f92\" (UID: \"e1dcd822-a886-418d-8f63-6b7daa823f92\") "
Nov 02 12:39:08.206431 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.206405    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-openshift-ca" pod "e1dcd822-a886-418d-8f63-6b7daa823f92" (UID: "e1dcd822-a886-418d-8f63-6b7daa823f92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:08.208149 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.208126    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-xks-ca" pod "e1dcd822-a886-418d-8f63-6b7daa823f92" (UID: "e1dcd822-a886-418d-8f63-6b7daa823f92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:39:08.208282 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.208261    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1dcd822-a886-418d-8f63-6b7daa823f92-kube-api-access-gcdtx" pod "e1dcd822-a886-418d-8f63-6b7daa823f92" (UID: "e1dcd822-a886-418d-8f63-6b7daa823f92"). InnerVolumeSpecName "kube-api-access-gcdtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:08.307787 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.307741    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcdtx\" (UniqueName: \"kubernetes.io/projected/e1dcd822-a886-418d-8f63-6b7daa823f92-kube-api-access-gcdtx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:39:08.307787 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.307781    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:39:08.307787 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.307796    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1dcd822-a886-418d-8f63-6b7daa823f92-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:39:08.968928 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.968894    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"
Nov 02 12:39:08.969165 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.968922    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q" event={"ID":"e1dcd822-a886-418d-8f63-6b7daa823f92","Type":"ContainerDied","Data":"40d53946ff5197f15d4bd7c4516d5cde28da0b30e8eaa9c0c853878594c26c38"}
Nov 02 12:39:08.969165 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:08.968987    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40d53946ff5197f15d4bd7c4516d5cde28da0b30e8eaa9c0c853878594c26c38"
Nov 02 12:39:09.367107 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:09.367077    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:10.949814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:10.949782    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:12.572089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:12.572055    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:14.302628 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:14.302589    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:14.820010 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:39:14.819959    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:39:16.080459 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:16.080422    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:16.442885 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:16.442821    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:16.576505 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:16.576467    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:17.802222 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:17.802178    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fx2tt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:39:17.802882 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:17.802842    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:39:18.037108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:18.037046    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:19.267698 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.267653    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"]
Nov 02 12:39:19.268415 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.268387    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e1dcd822-a886-418d-8f63-6b7daa823f92" containerName="revoke-keys"
Nov 02 12:39:19.268415 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.268413    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1dcd822-a886-418d-8f63-6b7daa823f92" containerName="revoke-keys"
Nov 02 12:39:19.268639 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.268532    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1dcd822-a886-418d-8f63-6b7daa823f92" containerName="revoke-keys"
Nov 02 12:39:19.272520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.272501    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.290208 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.290178    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5514e5\"/\"distinct-2-w0-5514e5-kserve-self-signed-certs\""
Nov 02 12:39:19.292884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.292855    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"]
Nov 02 12:39:19.342124 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.342076    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:19.354179 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.354137    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tls-certs\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.354179 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.354182    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-dshm\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.354436 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.354314    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tmp-dir\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.354436 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.354342    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-home\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.354552 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.354456    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-model-cache\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.354552 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.354522    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tfhq\" (UniqueName: \"kubernetes.io/projected/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-kube-api-access-4tfhq\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.456796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.456753    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tmp-dir\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.457079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.457058    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-home\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.457291 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.457272    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-model-cache\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.457435 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.457412    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tmp-dir\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.457625 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.457435    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfhq\" (UniqueName: \"kubernetes.io/projected/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-kube-api-access-4tfhq\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.457844 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.457828    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tls-certs\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.457974 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.457961    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-dshm\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.458226 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.458068    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-home\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.458351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.458277    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-model-cache\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.469124 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.469090    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-dshm\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.474597 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.474558    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tls-certs\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.491683 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.491654    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tfhq\" (UniqueName: \"kubernetes.io/projected/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-kube-api-access-4tfhq\") pod \"distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") " pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.587361 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.587270    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:19.742223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.742188    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:19.891866 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:19.891842    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"]
Nov 02 12:39:19.893439 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:39:19.893407    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f25ccf0_beac_4f85_b0ac_b08d8a9591ab.slice/crio-b01809c224065bb22d0990e885aca5694d9e7295aa54eada9a3f6628664d6b05 WatchSource:0}: Error finding container b01809c224065bb22d0990e885aca5694d9e7295aa54eada9a3f6628664d6b05: Status 404 returned error can't find the container with id b01809c224065bb22d0990e885aca5694d9e7295aa54eada9a3f6628664d6b05
Nov 02 12:39:20.049413 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:20.049375    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" event={"ID":"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab","Type":"ContainerStarted","Data":"2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370"}
Nov 02 12:39:20.049596 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:20.049424    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" event={"ID":"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab","Type":"ContainerStarted","Data":"b01809c224065bb22d0990e885aca5694d9e7295aa54eada9a3f6628664d6b05"}
Nov 02 12:39:20.097026 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:20.096952    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podStartSLOduration=1.096937004 podStartE2EDuration="1.096937004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:20.094919946 +0000 UTC m=+706.201622798" watchObservedRunningTime="2026-11-02 12:39:20.096937004 +0000 UTC m=+706.203639869"
Nov 02 12:39:20.588320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:20.588283    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:21.341911 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:21.341874    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:22.956173 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:22.956139    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:24.840050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:24.839075    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:26.532945 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:26.532909    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:28.057500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:28.057469    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:28.732264 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:28.732217    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:29.425971 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:29.425931    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:29.587742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:29.587708    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:29.600874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:29.600845    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:29.646728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:29.646689    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:30.133293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:30.133263    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:39:31.334424 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:31.334392    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:32.954326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:32.954294    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:34.007322 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:34.007280    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:34.629946 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:34.629915    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:36.246812 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:36.246762    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:37.767038 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:37.766958    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:37.880798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:37.880757    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:39.570183 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:39.570151    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:40.546891 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:40.546865    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:41.328925 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:41.328891    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:42.357591 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.357559    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"]
Nov 02 12:39:42.362854 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.362834    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.380816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.380791    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-5514e5-openshift-default-dockercfg-7v87s\""
Nov 02 12:39:42.399426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.399398    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"]
Nov 02 12:39:42.482206 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.482170    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.482421 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.482232    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fkw\" (UniqueName: \"kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-kube-api-access-22fkw\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.482421 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.482347    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.482421 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.482377    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.482421 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.482419    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.482589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.482487    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.482589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.482537    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.482589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.482571    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.482743 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.482627    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.584344 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.584312    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.584545 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.584357    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.584545 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.584483    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.584665 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.584555    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.584665 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.584616    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fkw\" (UniqueName: \"kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-kube-api-access-22fkw\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.584773 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.584671    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.584773 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.584704    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.584773 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.584743    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.584934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.584807    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.585009 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.584938    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.585075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.585029    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.585507 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.585479    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.585616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.585512    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.585616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.585533    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.587058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.587037    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.587269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.587250    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.610953 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.610876    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.611111 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.610948    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fkw\" (UniqueName: \"kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-kube-api-access-22fkw\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.678284 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.678247    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:42.795669 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.795632    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"]
Nov 02 12:39:42.800487 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.800460    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:42.828467 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.828431    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"]
Nov 02 12:39:42.990339 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.990296    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:42.990530 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.990354    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:42.990530 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.990458    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftl2\" (UniqueName: \"kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-kube-api-access-tftl2\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:42.990530 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.990508    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:42.990699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.990574    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:42.990759 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.990742    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:42.990817 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.990784    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:42.990889 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.990874    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:42.990957 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:42.990941    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.001872 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:39:43.001841    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5c93ad_afcd_4b72_b3ab_a04045b4746c.slice/crio-1edbb8867b1a99514d283e9538b39ef5c3a5749cf5f8a3a873ca3de5a5fa0b70 WatchSource:0}: Error finding container 1edbb8867b1a99514d283e9538b39ef5c3a5749cf5f8a3a873ca3de5a5fa0b70: Status 404 returned error can't find the container with id 1edbb8867b1a99514d283e9538b39ef5c3a5749cf5f8a3a873ca3de5a5fa0b70
Nov 02 12:39:43.002318 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.002274    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"]
Nov 02 12:39:43.004171 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.004139    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:39:43.004259 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.004204    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:39:43.004259 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.004233    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:39:43.047578 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.047546    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:43.092253 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.092219    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.092467 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.092277    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftl2\" (UniqueName: \"kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-kube-api-access-tftl2\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.092467 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.092330    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.092467 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.092357    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.092632 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.092479    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.092632 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.092509    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.092632 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.092555    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.092632 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.092616    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.092853 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.092679    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.093115 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.093069    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.093279 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.093262    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.093941 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.093904    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.104593 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.094547    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.104593 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.094609    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.104593 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.095552    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.104593 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.097157    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.118827 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.118799    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.119506 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.119481    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftl2\" (UniqueName: \"kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-kube-api-access-tftl2\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.121183 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.121139    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:43.210572 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.210325    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" event={"ID":"fc5c93ad-afcd-4b72-b3ab-a04045b4746c","Type":"ContainerStarted","Data":"1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c"}
Nov 02 12:39:43.210572 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.210389    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" event={"ID":"fc5c93ad-afcd-4b72-b3ab-a04045b4746c","Type":"ContainerStarted","Data":"1edbb8867b1a99514d283e9538b39ef5c3a5749cf5f8a3a873ca3de5a5fa0b70"}
Nov 02 12:39:43.259257 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.259128    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" podStartSLOduration=1.259107228 podStartE2EDuration="1.259107228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:43.257798198 +0000 UTC m=+729.364501051" watchObservedRunningTime="2026-11-02 12:39:43.259107228 +0000 UTC m=+729.365810080"
Nov 02 12:39:43.452740 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.452704    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"]
Nov 02 12:39:43.454096 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:39:43.454065    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod729df3ad_162a_47b5_9d93_351ec4ef3fd8.slice/crio-ceacc54baa00d8583e2c81894b0733b2f9cd513ecdd3a83c994cbacff44e5f6e WatchSource:0}: Error finding container ceacc54baa00d8583e2c81894b0733b2f9cd513ecdd3a83c994cbacff44e5f6e: Status 404 returned error can't find the container with id ceacc54baa00d8583e2c81894b0733b2f9cd513ecdd3a83c994cbacff44e5f6e
Nov 02 12:39:43.456641 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.456609    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:39:43.456703 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.456680    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:39:43.456783 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.456711    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:39:43.514029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.513906    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"]
Nov 02 12:39:43.520718 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.520685    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:43.533030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.532958    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"]
Nov 02 12:39:43.537668 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.537639    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:39:43.537948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.537923    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:39:43.678921 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.678875    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:43.716742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.716701    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdcjt\" (UniqueName: \"kubernetes.io/projected/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-kube-api-access-sdcjt\") pod \"maas-api-revoke-sub-f4b60761-8qj58\" (UID: \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:43.716907 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.716783    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b60761-8qj58\" (UID: \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:43.716907 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.716893    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b60761-8qj58\" (UID: \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:43.818332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.818235    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b60761-8qj58\" (UID: \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:43.818499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.818448    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdcjt\" (UniqueName: \"kubernetes.io/projected/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-kube-api-access-sdcjt\") pod \"maas-api-revoke-sub-f4b60761-8qj58\" (UID: \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:43.818499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.818483    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b60761-8qj58\" (UID: \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:43.819280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.819255    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b60761-8qj58\" (UID: \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:43.821464 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.821436    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b60761-8qj58\" (UID: \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:43.844541 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.844510    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdcjt\" (UniqueName: \"kubernetes.io/projected/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-kube-api-access-sdcjt\") pod \"maas-api-revoke-sub-f4b60761-8qj58\" (UID: \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:43.850009 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:43.849969    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:44.208476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:44.208424    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"]
Nov 02 12:39:44.214832 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:39:44.214801    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ecddb26_4e40_4c73_a1cb_01d157c8e8ce.slice/crio-f53f5cbaf3e321735ae8463f1ae520b1ebab1b3161e7b86b06b898e2ec93ce44 WatchSource:0}: Error finding container f53f5cbaf3e321735ae8463f1ae520b1ebab1b3161e7b86b06b898e2ec93ce44: Status 404 returned error can't find the container with id f53f5cbaf3e321735ae8463f1ae520b1ebab1b3161e7b86b06b898e2ec93ce44
Nov 02 12:39:44.219274 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:44.219240    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" event={"ID":"729df3ad-162a-47b5-9d93-351ec4ef3fd8","Type":"ContainerStarted","Data":"5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef"}
Nov 02 12:39:44.219405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:44.219301    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" event={"ID":"729df3ad-162a-47b5-9d93-351ec4ef3fd8","Type":"ContainerStarted","Data":"ceacc54baa00d8583e2c81894b0733b2f9cd513ecdd3a83c994cbacff44e5f6e"}
Nov 02 12:39:44.302043 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:44.300832    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" podStartSLOduration=2.300807932 podStartE2EDuration="2.300807932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:44.299514465 +0000 UTC m=+730.406217317" watchObservedRunningTime="2026-11-02 12:39:44.300807932 +0000 UTC m=+730.407510784"
Nov 02 12:39:44.680382 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:44.680342    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 12:39:44.680779 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:44.680417    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" podUID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 12:39:44.861108 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:39:44.861070    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:39:44.963987 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:44.963906    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:45.135019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:45.134166    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:45.140029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:45.139119    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:39:45.140029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:45.139188    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" podUID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 12:39:45.242118 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:45.242025    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58" event={"ID":"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce","Type":"ContainerStarted","Data":"937dc1767ceb90b37b8b323caeb074138e5df196c9b2a42eae754eef3ee85ee0"}
Nov 02 12:39:45.242118 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:45.242077    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58" event={"ID":"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce","Type":"ContainerStarted","Data":"f53f5cbaf3e321735ae8463f1ae520b1ebab1b3161e7b86b06b898e2ec93ce44"}
Nov 02 12:39:45.679183 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:45.679149    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 12:39:45.679378 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:45.679213    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" podUID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 12:39:46.126191 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:46.126155    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:39:46.126798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:46.126764    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" podUID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 12:39:46.249961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:46.249927    5879 generic.go:356] "Generic (PLEG): container finished" podID="0ecddb26-4e40-4c73-a1cb-01d157c8e8ce" containerID="937dc1767ceb90b37b8b323caeb074138e5df196c9b2a42eae754eef3ee85ee0" exitCode=0
Nov 02 12:39:46.250253 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:46.250225    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58" event={"ID":"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce","Type":"ContainerDied","Data":"937dc1767ceb90b37b8b323caeb074138e5df196c9b2a42eae754eef3ee85ee0"}
Nov 02 12:39:46.679529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:46.679496    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 12:39:46.679707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:46.679560    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" podUID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 12:39:46.829021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:46.828127    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:47.136691 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.126136    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:39:47.136691 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.126194    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" podUID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 12:39:47.576086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.574883    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:47.622094 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.621984    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:47.679557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.679525    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-kube-api-access-sdcjt\" (UniqueName: \"kubernetes.io/projected/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-kube-api-access-sdcjt\") pod \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\" (UID: \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\") "
Nov 02 12:39:47.679964 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.679940    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-xks-ca\") pod \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\" (UID: \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\") "
Nov 02 12:39:47.680139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.680122    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-openshift-ca\") pod \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\" (UID: \"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce\") "
Nov 02 12:39:47.682103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.682068    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-openshift-ca" pod "0ecddb26-4e40-4c73-a1cb-01d157c8e8ce" (UID: "0ecddb26-4e40-4c73-a1cb-01d157c8e8ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:47.694548 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.694517    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 12:39:47.694814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.694772    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" podUID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 12:39:47.711768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.711720    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-xks-ca" pod "0ecddb26-4e40-4c73-a1cb-01d157c8e8ce" (UID: "0ecddb26-4e40-4c73-a1cb-01d157c8e8ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:39:47.712164 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.712139    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-kube-api-access-sdcjt" pod "0ecddb26-4e40-4c73-a1cb-01d157c8e8ce" (UID: "0ecddb26-4e40-4c73-a1cb-01d157c8e8ce"). InnerVolumeSpecName "kube-api-access-sdcjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:47.783514 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.783479    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdcjt\" (UniqueName: \"kubernetes.io/projected/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-kube-api-access-sdcjt\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:39:47.783750 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.783732    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:39:47.783902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:47.783888    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:39:48.132184 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:48.132147    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:39:48.132528 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:48.132494    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" podUID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 12:39:48.263583 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:48.263549    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58" event={"ID":"0ecddb26-4e40-4c73-a1cb-01d157c8e8ce","Type":"ContainerDied","Data":"f53f5cbaf3e321735ae8463f1ae520b1ebab1b3161e7b86b06b898e2ec93ce44"}
Nov 02 12:39:48.263977 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:48.263598    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f53f5cbaf3e321735ae8463f1ae520b1ebab1b3161e7b86b06b898e2ec93ce44"
Nov 02 12:39:48.263977 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:48.263652    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"
Nov 02 12:39:48.680018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:48.679637    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 12:39:48.680018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:48.679714    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" podUID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 12:39:48.836533 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:48.836497    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:49.123154 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:49.123118    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:39:49.123453 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:49.123424    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" podUID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 12:39:49.679340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:49.679299    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 12:39:49.679839 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:49.679366    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" podUID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 12:39:50.122589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:50.122542    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:39:50.122805 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:50.122617    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" podUID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 12:39:50.576050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:50.575940    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:51.122464 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:51.122426    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 12:39:51.122902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:51.122505    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" podUID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 12:39:51.721218 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:51.721174    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:39:51.721423 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:51.721249    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" podUID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:39:52.002210 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:52.002130    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:52.002448 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:52.002429    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:52.003089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:52.003069    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 12:39:52.163449 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:52.163418    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:52.743462 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:52.743433    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:52.743926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:52.743893    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:52.744711 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:52.744678    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 12:39:53.828363 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:53.828327    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:55.355674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:55.355641    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:55.823588 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:55.823554    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"]
Nov 02 12:39:55.824268 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:55.824251    5879 cpu_manager.go:414] "containerMap: removing container" podUID="0ecddb26-4e40-4c73-a1cb-01d157c8e8ce" containerName="revoke-keys"
Nov 02 12:39:55.824326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:55.824272    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ecddb26-4e40-4c73-a1cb-01d157c8e8ce" containerName="revoke-keys"
Nov 02 12:39:55.824432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:55.824422    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ecddb26-4e40-4c73-a1cb-01d157c8e8ce" containerName="revoke-keys"
Nov 02 12:39:55.828198 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:55.828182    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:39:55.848220 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:55.848186    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"]
Nov 02 12:39:55.860602 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:55.860563    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:39:55.860602 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:55.860585    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:39:55.923980 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:55.923936    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d0404ed-v4lm7\" (UID: \"d46e8118-e52c-4a9b-80ab-fae8692e228d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:39:55.924184 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:55.924111    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5kc\" (UniqueName: \"kubernetes.io/projected/d46e8118-e52c-4a9b-80ab-fae8692e228d-kube-api-access-wr5kc\") pod \"maas-api-revoke-sub-5d0404ed-v4lm7\" (UID: \"d46e8118-e52c-4a9b-80ab-fae8692e228d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:39:55.924285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:55.924270    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d0404ed-v4lm7\" (UID: \"d46e8118-e52c-4a9b-80ab-fae8692e228d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:39:56.025914 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:56.025879    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d0404ed-v4lm7\" (UID: \"d46e8118-e52c-4a9b-80ab-fae8692e228d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:39:56.026104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:56.025941    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d0404ed-v4lm7\" (UID: \"d46e8118-e52c-4a9b-80ab-fae8692e228d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:39:56.026104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:56.026070    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5kc\" (UniqueName: \"kubernetes.io/projected/d46e8118-e52c-4a9b-80ab-fae8692e228d-kube-api-access-wr5kc\") pod \"maas-api-revoke-sub-5d0404ed-v4lm7\" (UID: \"d46e8118-e52c-4a9b-80ab-fae8692e228d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:39:56.026585 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:56.026564    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d0404ed-v4lm7\" (UID: \"d46e8118-e52c-4a9b-80ab-fae8692e228d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:39:56.028574 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:56.028556    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d0404ed-v4lm7\" (UID: \"d46e8118-e52c-4a9b-80ab-fae8692e228d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:39:56.051615 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:56.051577    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5kc\" (UniqueName: \"kubernetes.io/projected/d46e8118-e52c-4a9b-80ab-fae8692e228d-kube-api-access-wr5kc\") pod \"maas-api-revoke-sub-5d0404ed-v4lm7\" (UID: \"d46e8118-e52c-4a9b-80ab-fae8692e228d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:39:56.142333 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:56.142299    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:39:56.489271 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:39:56.489234    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46e8118_e52c_4a9b_80ab_fae8692e228d.slice/crio-24b6bad0faf2d9d3bfb21c3e261bb5fbc62f4bffe28bdcf2374389f3859c515f WatchSource:0}: Error finding container 24b6bad0faf2d9d3bfb21c3e261bb5fbc62f4bffe28bdcf2374389f3859c515f: Status 404 returned error can't find the container with id 24b6bad0faf2d9d3bfb21c3e261bb5fbc62f4bffe28bdcf2374389f3859c515f
Nov 02 12:39:56.489658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:56.489468    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"]
Nov 02 12:39:57.026117 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:57.026081    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:57.339641 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:57.339549    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7" event={"ID":"d46e8118-e52c-4a9b-80ab-fae8692e228d","Type":"ContainerStarted","Data":"cc3cc804e8a23da01acccbea85c7b9122bdeeb1f1f9cf9c2840e79563094e95f"}
Nov 02 12:39:57.339818 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:57.339654    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7" event={"ID":"d46e8118-e52c-4a9b-80ab-fae8692e228d","Type":"ContainerStarted","Data":"24b6bad0faf2d9d3bfb21c3e261bb5fbc62f4bffe28bdcf2374389f3859c515f"}
Nov 02 12:39:57.383101 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:57.383050    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7" podStartSLOduration=2.383035808 podStartE2EDuration="2.383035808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:39:57.382090547 +0000 UTC m=+743.488793400" watchObservedRunningTime="2026-11-02 12:39:57.383035808 +0000 UTC m=+743.489738658"
Nov 02 12:39:58.347944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:58.347911    5879 generic.go:356] "Generic (PLEG): container finished" podID="d46e8118-e52c-4a9b-80ab-fae8692e228d" containerID="cc3cc804e8a23da01acccbea85c7b9122bdeeb1f1f9cf9c2840e79563094e95f" exitCode=0
Nov 02 12:39:58.348351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:58.348012    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7" event={"ID":"d46e8118-e52c-4a9b-80ab-fae8692e228d","Type":"ContainerDied","Data":"cc3cc804e8a23da01acccbea85c7b9122bdeeb1f1f9cf9c2840e79563094e95f"}
Nov 02 12:39:58.654669 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:58.654631    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:39:59.506973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:59.506945    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:39:59.573035 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:59.572986    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46e8118-e52c-4a9b-80ab-fae8692e228d-kube-api-access-wr5kc\" (UniqueName: \"kubernetes.io/projected/d46e8118-e52c-4a9b-80ab-fae8692e228d-kube-api-access-wr5kc\") pod \"d46e8118-e52c-4a9b-80ab-fae8692e228d\" (UID: \"d46e8118-e52c-4a9b-80ab-fae8692e228d\") "
Nov 02 12:39:59.573230 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:59.573071    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-openshift-ca\") pod \"d46e8118-e52c-4a9b-80ab-fae8692e228d\" (UID: \"d46e8118-e52c-4a9b-80ab-fae8692e228d\") "
Nov 02 12:39:59.573319 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:59.573300    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-xks-ca\") pod \"d46e8118-e52c-4a9b-80ab-fae8692e228d\" (UID: \"d46e8118-e52c-4a9b-80ab-fae8692e228d\") "
Nov 02 12:39:59.573429 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:59.573406    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-openshift-ca" pod "d46e8118-e52c-4a9b-80ab-fae8692e228d" (UID: "d46e8118-e52c-4a9b-80ab-fae8692e228d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:39:59.573807 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:59.573765    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:39:59.575321 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:59.575291    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46e8118-e52c-4a9b-80ab-fae8692e228d-kube-api-access-wr5kc" pod "d46e8118-e52c-4a9b-80ab-fae8692e228d" (UID: "d46e8118-e52c-4a9b-80ab-fae8692e228d"). InnerVolumeSpecName "kube-api-access-wr5kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:39:59.575436 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:59.575418    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-xks-ca" pod "d46e8118-e52c-4a9b-80ab-fae8692e228d" (UID: "d46e8118-e52c-4a9b-80ab-fae8692e228d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:39:59.674927 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:59.674889    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46e8118-e52c-4a9b-80ab-fae8692e228d-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:39:59.674927 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:39:59.674919    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr5kc\" (UniqueName: \"kubernetes.io/projected/d46e8118-e52c-4a9b-80ab-fae8692e228d-kube-api-access-wr5kc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:00.252387 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:00.252348    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:00.361782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:00.361699    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7" event={"ID":"d46e8118-e52c-4a9b-80ab-fae8692e228d","Type":"ContainerDied","Data":"24b6bad0faf2d9d3bfb21c3e261bb5fbc62f4bffe28bdcf2374389f3859c515f"}
Nov 02 12:40:00.361782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:00.361754    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24b6bad0faf2d9d3bfb21c3e261bb5fbc62f4bffe28bdcf2374389f3859c515f"
Nov 02 12:40:00.361782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:00.361711    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"
Nov 02 12:40:01.779028 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.778980    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"]
Nov 02 12:40:01.779723 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.779699    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d46e8118-e52c-4a9b-80ab-fae8692e228d" containerName="revoke-keys"
Nov 02 12:40:01.779723 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.779723    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46e8118-e52c-4a9b-80ab-fae8692e228d" containerName="revoke-keys"
Nov 02 12:40:01.779888 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.779862    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46e8118-e52c-4a9b-80ab-fae8692e228d" containerName="revoke-keys"
Nov 02 12:40:01.783588 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.783562    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:01.818185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.818151    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"]
Nov 02 12:40:01.900586 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.900530    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:01.900753 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.900588    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:01.900812 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.900753    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:01.900883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.900853    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7m26\" (UniqueName: \"kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-kube-api-access-l7m26\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:01.900933 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.900888    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a30efd-4594-4665-b620-61748d33306f-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:01.901021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.900961    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:01.901079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.901021    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:01.901079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.901045    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a30efd-4594-4665-b620-61748d33306f-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:01.901079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.901063    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:01.943777 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:01.943745    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:02.001956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.001923    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7m26\" (UniqueName: \"kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-kube-api-access-l7m26\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.002142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.001975    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a30efd-4594-4665-b620-61748d33306f-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.002142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.002037    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.002142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.002065    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.002142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.002093    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a30efd-4594-4665-b620-61748d33306f-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.002142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.002125    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.002429 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.002162    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.002429 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.002188    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.002429 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.002302    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.002699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.002673    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-credential-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.002826 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.002794    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a30efd-4594-4665-b620-61748d33306f-istiod-ca-cert\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.003237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.003133    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-data\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.004160 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.004100    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-socket\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.004276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.004175    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-certs\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.005383 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.005361    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a30efd-4594-4665-b620-61748d33306f-istio-podinfo\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.005490 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.005405    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-envoy\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.027574 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.027541    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-istio-token\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.027802 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.027779    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7m26\" (UniqueName: \"kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-kube-api-access-l7m26\") pod \"e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") " pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.102060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.100558    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:02.483591 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.483556    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"]
Nov 02 12:40:02.485404 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:40:02.485366    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a30efd_4594_4665_b620_61748d33306f.slice/crio-32e5a82353867740050cf3b277a05459fbb332999dd4c78f484bae854b994f03 WatchSource:0}: Error finding container 32e5a82353867740050cf3b277a05459fbb332999dd4c78f484bae854b994f03: Status 404 returned error can't find the container with id 32e5a82353867740050cf3b277a05459fbb332999dd4c78f484bae854b994f03
Nov 02 12:40:02.488400 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.488354    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:40:02.488529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.488449    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:40:02.488529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:02.488489    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:40:03.413018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:03.412290    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" event={"ID":"d5a30efd-4594-4665-b620-61748d33306f","Type":"ContainerStarted","Data":"7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677"}
Nov 02 12:40:03.413018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:03.412362    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" event={"ID":"d5a30efd-4594-4665-b620-61748d33306f","Type":"ContainerStarted","Data":"32e5a82353867740050cf3b277a05459fbb332999dd4c78f484bae854b994f03"}
Nov 02 12:40:03.503027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:03.501487    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" podStartSLOduration=2.50146494 podStartE2EDuration="2.50146494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:03.487490069 +0000 UTC m=+749.594192932" watchObservedRunningTime="2026-11-02 12:40:03.50146494 +0000 UTC m=+749.608167792"
Nov 02 12:40:04.107073 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:04.107032    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:04.109113 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:04.109086    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:04.109325 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:04.109300    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" podUID="d5a30efd-4594-4665-b620-61748d33306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 12:40:04.260022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:04.255959    5879 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:04.518682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:04.518650    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:05.106838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:05.106379    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:05.106838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:05.106444    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" podUID="d5a30efd-4594-4665-b620-61748d33306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 12:40:05.315696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:05.315646    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:06.106022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:06.104460    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:06.106022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:06.104539    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" podUID="d5a30efd-4594-4665-b620-61748d33306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 12:40:06.328044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:06.327982    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:07.111429 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.111387    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:07.111930 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.111453    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" podUID="d5a30efd-4594-4665-b620-61748d33306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 12:40:07.610019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.605100    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"]
Nov 02 12:40:07.612490 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.612464    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:07.627263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.627232    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"]
Nov 02 12:40:07.636487 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.636452    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:40:07.636640 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.636520    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:40:07.708018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.707599    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7cqz\" (UniqueName: \"kubernetes.io/projected/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-kube-api-access-w7cqz\") pod \"maas-api-revoke-sub-c3173f45-bxrf2\" (UID: \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:07.708018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.707802    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3173f45-bxrf2\" (UID: \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:07.708277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.708081    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3173f45-bxrf2\" (UID: \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:07.809388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.809350    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3173f45-bxrf2\" (UID: \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:07.809660 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.809501    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3173f45-bxrf2\" (UID: \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:07.809660 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.809582    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7cqz\" (UniqueName: \"kubernetes.io/projected/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-kube-api-access-w7cqz\") pod \"maas-api-revoke-sub-c3173f45-bxrf2\" (UID: \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:07.810378 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.810305    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3173f45-bxrf2\" (UID: \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:07.812094 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.812071    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3173f45-bxrf2\" (UID: \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:07.834935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.834906    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7cqz\" (UniqueName: \"kubernetes.io/projected/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-kube-api-access-w7cqz\") pod \"maas-api-revoke-sub-c3173f45-bxrf2\" (UID: \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:07.931252 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:07.931221    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:08.112466 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:08.112419    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:08.112973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:08.112484    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" podUID="d5a30efd-4594-4665-b620-61748d33306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 12:40:08.171475 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:08.171438    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:08.269589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:08.269560    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"]
Nov 02 12:40:08.274410 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:40:08.273828    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea33eed1_d3b6_40f9_8d5f_6542bd424e6a.slice/crio-c0395e5739d58c818c7641dab20f7b744f19d4f1a164b93f2cd6f170e8f43273 WatchSource:0}: Error finding container c0395e5739d58c818c7641dab20f7b744f19d4f1a164b93f2cd6f170e8f43273: Status 404 returned error can't find the container with id c0395e5739d58c818c7641dab20f7b744f19d4f1a164b93f2cd6f170e8f43273
Nov 02 12:40:08.449901 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:08.449864    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2" event={"ID":"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a","Type":"ContainerStarted","Data":"c0395e5739d58c818c7641dab20f7b744f19d4f1a164b93f2cd6f170e8f43273"}
Nov 02 12:40:09.105020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:09.104546    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:09.105020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:09.104618    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" podUID="d5a30efd-4594-4665-b620-61748d33306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 12:40:09.457579 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:09.457546    5879 generic.go:356] "Generic (PLEG): container finished" podID="ea33eed1-d3b6-40f9-8d5f-6542bd424e6a" containerID="2d7f4964acf9bfd577dcd3ad680710700909aa93cf0e588d388596f79fa6f4d6" exitCode=0
Nov 02 12:40:09.458131 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:09.457608    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2" event={"ID":"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a","Type":"ContainerDied","Data":"2d7f4964acf9bfd577dcd3ad680710700909aa93cf0e588d388596f79fa6f4d6"}
Nov 02 12:40:09.891207 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:09.891170    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:10.102765 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:10.102731    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 12:40:10.102945 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:10.102792    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" podUID="d5a30efd-4594-4665-b620-61748d33306f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 12:40:10.651853 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:10.651824    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:10.747570 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:10.747489    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-openshift-ca\") pod \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\" (UID: \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\") "
Nov 02 12:40:10.747570 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:10.747547    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-xks-ca\") pod \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\" (UID: \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\") "
Nov 02 12:40:10.747749 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:10.747638    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-kube-api-access-w7cqz\" (UniqueName: \"kubernetes.io/projected/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-kube-api-access-w7cqz\") pod \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\" (UID: \"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a\") "
Nov 02 12:40:10.747936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:10.747911    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-openshift-ca" pod "ea33eed1-d3b6-40f9-8d5f-6542bd424e6a" (UID: "ea33eed1-d3b6-40f9-8d5f-6542bd424e6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:10.748158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:10.748135    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:10.749808 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:10.749779    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-kube-api-access-w7cqz" pod "ea33eed1-d3b6-40f9-8d5f-6542bd424e6a" (UID: "ea33eed1-d3b6-40f9-8d5f-6542bd424e6a"). InnerVolumeSpecName "kube-api-access-w7cqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:10.749902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:10.749805    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-xks-ca" pod "ea33eed1-d3b6-40f9-8d5f-6542bd424e6a" (UID: "ea33eed1-d3b6-40f9-8d5f-6542bd424e6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:40:10.849609 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:10.849576    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:10.849609 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:10.849605    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7cqz\" (UniqueName: \"kubernetes.io/projected/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a-kube-api-access-w7cqz\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:11.107692 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:11.107615    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:11.108027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:11.108007    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:11.108725 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:11.108705    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 12:40:11.222021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:11.221895    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:11.474595 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:11.474565    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"
Nov 02 12:40:11.474595 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:11.474563    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2" event={"ID":"ea33eed1-d3b6-40f9-8d5f-6542bd424e6a","Type":"ContainerDied","Data":"c0395e5739d58c818c7641dab20f7b744f19d4f1a164b93f2cd6f170e8f43273"}
Nov 02 12:40:11.474803 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:11.474625    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0395e5739d58c818c7641dab20f7b744f19d4f1a164b93f2cd6f170e8f43273"
Nov 02 12:40:11.589492 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:11.589460    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:11.784166 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:11.784080    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:13.246803 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:13.246759    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:14.822361 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:40:14.822324    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:40:15.036089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:15.036056    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:16.549853 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:16.549822    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:18.239523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:18.239490    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:19.956500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:19.956410    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:20.292965 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.291971    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"]
Nov 02 12:40:20.299103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.299076    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ea33eed1-d3b6-40f9-8d5f-6542bd424e6a" containerName="revoke-keys"
Nov 02 12:40:20.299254 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.299239    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea33eed1-d3b6-40f9-8d5f-6542bd424e6a" containerName="revoke-keys"
Nov 02 12:40:20.299553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.299536    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea33eed1-d3b6-40f9-8d5f-6542bd424e6a" containerName="revoke-keys"
Nov 02 12:40:20.326080 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.323959    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"]
Nov 02 12:40:20.326342 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.324136    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:20.347053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.347029    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:40:20.366531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.366500    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:40:20.434301 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.410157    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6572f7e0-56m9b\" (UID: \"2a831c41-4a83-4837-9c9f-a93452551bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:20.434713 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.434677    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ldzn\" (UniqueName: \"kubernetes.io/projected/2a831c41-4a83-4837-9c9f-a93452551bb0-kube-api-access-8ldzn\") pod \"maas-api-revoke-sub-6572f7e0-56m9b\" (UID: \"2a831c41-4a83-4837-9c9f-a93452551bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:20.434980 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.434960    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6572f7e0-56m9b\" (UID: \"2a831c41-4a83-4837-9c9f-a93452551bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:20.564074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.563636    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6572f7e0-56m9b\" (UID: \"2a831c41-4a83-4837-9c9f-a93452551bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:20.564074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.563711    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ldzn\" (UniqueName: \"kubernetes.io/projected/2a831c41-4a83-4837-9c9f-a93452551bb0-kube-api-access-8ldzn\") pod \"maas-api-revoke-sub-6572f7e0-56m9b\" (UID: \"2a831c41-4a83-4837-9c9f-a93452551bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:20.564074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.563816    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6572f7e0-56m9b\" (UID: \"2a831c41-4a83-4837-9c9f-a93452551bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:20.567657 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.566684    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6572f7e0-56m9b\" (UID: \"2a831c41-4a83-4837-9c9f-a93452551bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:20.576847 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.576816    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6572f7e0-56m9b\" (UID: \"2a831c41-4a83-4837-9c9f-a93452551bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:20.638944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.638914    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ldzn\" (UniqueName: \"kubernetes.io/projected/2a831c41-4a83-4837-9c9f-a93452551bb0-kube-api-access-8ldzn\") pod \"maas-api-revoke-sub-6572f7e0-56m9b\" (UID: \"2a831c41-4a83-4837-9c9f-a93452551bb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:20.671249 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:20.671212    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:21.337031 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:21.333112    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"]
Nov 02 12:40:21.590377 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:21.590214    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b" event={"ID":"2a831c41-4a83-4837-9c9f-a93452551bb0","Type":"ContainerStarted","Data":"1d18de008b8abac5dcbf713139eebe6f6d544da57911c1e778f3e40cd866adab"}
Nov 02 12:40:22.464119 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:22.464080    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:22.598973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:22.598944    5879 generic.go:356] "Generic (PLEG): container finished" podID="2a831c41-4a83-4837-9c9f-a93452551bb0" containerID="7ea6558e477e9aa10f7680598f6d17d4cc46ce880df6b90d8808acb83e7b82f6" exitCode=0
Nov 02 12:40:22.599160 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:22.599019    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b" event={"ID":"2a831c41-4a83-4837-9c9f-a93452551bb0","Type":"ContainerDied","Data":"7ea6558e477e9aa10f7680598f6d17d4cc46ce880df6b90d8808acb83e7b82f6"}
Nov 02 12:40:23.107407 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.107376    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"]
Nov 02 12:40:23.111244 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.111223    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:23.131799 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.131769    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"]
Nov 02 12:40:23.231633 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.231597    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d57be6-zc9ng\" (UID: \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:23.231831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.231640    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d57be6-zc9ng\" (UID: \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:23.231831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.231816    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fms47\" (UniqueName: \"kubernetes.io/projected/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-kube-api-access-fms47\") pod \"maas-api-revoke-sub-57d57be6-zc9ng\" (UID: \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:23.333555 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.333523    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fms47\" (UniqueName: \"kubernetes.io/projected/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-kube-api-access-fms47\") pod \"maas-api-revoke-sub-57d57be6-zc9ng\" (UID: \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:23.333737 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.333676    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d57be6-zc9ng\" (UID: \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:23.333737 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.333723    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d57be6-zc9ng\" (UID: \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:23.334345 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.334321    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d57be6-zc9ng\" (UID: \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:23.336379 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.336357    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d57be6-zc9ng\" (UID: \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:23.359965 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.359900    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fms47\" (UniqueName: \"kubernetes.io/projected/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-kube-api-access-fms47\") pod \"maas-api-revoke-sub-57d57be6-zc9ng\" (UID: \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:23.425712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.425680    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:23.737443 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:40:23.737405    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd1eabb_3ff8_4bab_8fcf_025e2d2e53d6.slice/crio-c20d6d33dda51a3e336119f94d9291c99529ef5c04287eb4cf87350ee155c4bc WatchSource:0}: Error finding container c20d6d33dda51a3e336119f94d9291c99529ef5c04287eb4cf87350ee155c4bc: Status 404 returned error can't find the container with id c20d6d33dda51a3e336119f94d9291c99529ef5c04287eb4cf87350ee155c4bc
Nov 02 12:40:23.738386 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.738354    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"]
Nov 02 12:40:23.770677 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.770653    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:23.939393 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.939312    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-openshift-ca\") pod \"2a831c41-4a83-4837-9c9f-a93452551bb0\" (UID: \"2a831c41-4a83-4837-9c9f-a93452551bb0\") "
Nov 02 12:40:23.939540 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.939503    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a831c41-4a83-4837-9c9f-a93452551bb0-kube-api-access-8ldzn\" (UniqueName: \"kubernetes.io/projected/2a831c41-4a83-4837-9c9f-a93452551bb0-kube-api-access-8ldzn\") pod \"2a831c41-4a83-4837-9c9f-a93452551bb0\" (UID: \"2a831c41-4a83-4837-9c9f-a93452551bb0\") "
Nov 02 12:40:23.939714 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.939680    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-xks-ca\") pod \"2a831c41-4a83-4837-9c9f-a93452551bb0\" (UID: \"2a831c41-4a83-4837-9c9f-a93452551bb0\") "
Nov 02 12:40:23.939876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.939741    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-openshift-ca" pod "2a831c41-4a83-4837-9c9f-a93452551bb0" (UID: "2a831c41-4a83-4837-9c9f-a93452551bb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:23.940180 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.940159    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:23.941975 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.941954    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-xks-ca" pod "2a831c41-4a83-4837-9c9f-a93452551bb0" (UID: "2a831c41-4a83-4837-9c9f-a93452551bb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:40:23.942299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:23.942270    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a831c41-4a83-4837-9c9f-a93452551bb0-kube-api-access-8ldzn" pod "2a831c41-4a83-4837-9c9f-a93452551bb0" (UID: "2a831c41-4a83-4837-9c9f-a93452551bb0"). InnerVolumeSpecName "kube-api-access-8ldzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:24.041690 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:24.041655    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a831c41-4a83-4837-9c9f-a93452551bb0-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:24.041690 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:24.041680    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ldzn\" (UniqueName: \"kubernetes.io/projected/2a831c41-4a83-4837-9c9f-a93452551bb0-kube-api-access-8ldzn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:24.163111 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:24.163072    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:24.613481 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:24.613389    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"
Nov 02 12:40:24.613638 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:24.613389    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b" event={"ID":"2a831c41-4a83-4837-9c9f-a93452551bb0","Type":"ContainerDied","Data":"1d18de008b8abac5dcbf713139eebe6f6d544da57911c1e778f3e40cd866adab"}
Nov 02 12:40:24.613638 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:24.613525    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d18de008b8abac5dcbf713139eebe6f6d544da57911c1e778f3e40cd866adab"
Nov 02 12:40:24.615269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:24.615242    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng" event={"ID":"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6","Type":"ContainerStarted","Data":"d7c0b7a6bb3313155642ff636bf33fe67aca04d4f19a85c232391bc0dbfab510"}
Nov 02 12:40:24.615416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:24.615291    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng" event={"ID":"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6","Type":"ContainerStarted","Data":"c20d6d33dda51a3e336119f94d9291c99529ef5c04287eb4cf87350ee155c4bc"}
Nov 02 12:40:24.658924 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:24.658871    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng" podStartSLOduration=1.65885796 podStartE2EDuration="1.65885796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:24.657520034 +0000 UTC m=+770.764222886" watchObservedRunningTime="2026-11-02 12:40:24.65885796 +0000 UTC m=+770.765560812"
Nov 02 12:40:25.624844 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:25.624803    5879 generic.go:356] "Generic (PLEG): container finished" podID="9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6" containerID="d7c0b7a6bb3313155642ff636bf33fe67aca04d4f19a85c232391bc0dbfab510" exitCode=0
Nov 02 12:40:25.625297 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:25.625007    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng" event={"ID":"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6","Type":"ContainerDied","Data":"d7c0b7a6bb3313155642ff636bf33fe67aca04d4f19a85c232391bc0dbfab510"}
Nov 02 12:40:25.861857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:25.861827    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:26.782834 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:26.782805    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:26.883644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:26.883571    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-openshift-ca\") pod \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\" (UID: \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\") "
Nov 02 12:40:26.883644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:26.883636    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-kube-api-access-fms47\" (UniqueName: \"kubernetes.io/projected/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-kube-api-access-fms47\") pod \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\" (UID: \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\") "
Nov 02 12:40:26.883875 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:26.883660    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-xks-ca\") pod \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\" (UID: \"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6\") "
Nov 02 12:40:26.883985 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:26.883895    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-openshift-ca" pod "9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6" (UID: "9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:26.884275 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:26.884259    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:26.886043 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:26.885976    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-xks-ca" pod "9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6" (UID: "9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:40:26.886406 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:26.886385    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-kube-api-access-fms47" pod "9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6" (UID: "9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6"). InnerVolumeSpecName "kube-api-access-fms47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:26.985022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:26.984963    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fms47\" (UniqueName: \"kubernetes.io/projected/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-kube-api-access-fms47\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:26.985022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:26.985020    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:27.520419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.520380    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"]
Nov 02 12:40:27.521283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.521264    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2a831c41-4a83-4837-9c9f-a93452551bb0" containerName="revoke-keys"
Nov 02 12:40:27.521352 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.521288    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a831c41-4a83-4837-9c9f-a93452551bb0" containerName="revoke-keys"
Nov 02 12:40:27.521388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.521350    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6" containerName="revoke-keys"
Nov 02 12:40:27.521388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.521359    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6" containerName="revoke-keys"
Nov 02 12:40:27.521490 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.521477    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6" containerName="revoke-keys"
Nov 02 12:40:27.521528 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.521501    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a831c41-4a83-4837-9c9f-a93452551bb0" containerName="revoke-keys"
Nov 02 12:40:27.525775 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.525752    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.544148 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.544116    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"]
Nov 02 12:40:27.549031 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.548972    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:27.553914 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.553877    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-95ee26\"/\"distinct-w2-95ee26-kserve-self-signed-certs\""
Nov 02 12:40:27.591585 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.591547    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjxlx\" (UniqueName: \"kubernetes.io/projected/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-kube-api-access-jjxlx\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.591755 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.591698    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tmp-dir\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.591815 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.591794    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-model-cache\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.591942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.591927    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-home\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.592073 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.592059    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-dshm\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.592137 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.592097    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tls-certs\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.640329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.640293    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"
Nov 02 12:40:27.640329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.640312    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng" event={"ID":"9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6","Type":"ContainerDied","Data":"c20d6d33dda51a3e336119f94d9291c99529ef5c04287eb4cf87350ee155c4bc"}
Nov 02 12:40:27.640553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.640374    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c20d6d33dda51a3e336119f94d9291c99529ef5c04287eb4cf87350ee155c4bc"
Nov 02 12:40:27.672350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.672287    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:27.692940 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.692908    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjxlx\" (UniqueName: \"kubernetes.io/projected/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-kube-api-access-jjxlx\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.693164 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.693030    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tmp-dir\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.693164 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.693080    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-model-cache\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.693294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.693233    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-home\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.693357 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.693308    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-dshm\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.693357 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.693347    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tls-certs\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.693664 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.693608    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-model-cache\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.693915 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.693769    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tmp-dir\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.696201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.696115    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-home\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.698288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.698260    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tls-certs\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.714015 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.713925    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-dshm\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.720423 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.719563    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjxlx\" (UniqueName: \"kubernetes.io/projected/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-kube-api-access-jjxlx\") pod \"distinct-w2-95ee26-kserve-69ff98695f-4947s\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") " pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:27.841273 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:27.841181    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:28.152503 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:40:28.152468    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91b3cc5f_ed9c_4662_9889_ee99a4a8c0da.slice/crio-6b9736b78a743fb981364d7842f862dd2e167b121b98e211428e9f3fd29b462f WatchSource:0}: Error finding container 6b9736b78a743fb981364d7842f862dd2e167b121b98e211428e9f3fd29b462f: Status 404 returned error can't find the container with id 6b9736b78a743fb981364d7842f862dd2e167b121b98e211428e9f3fd29b462f
Nov 02 12:40:28.154047 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:28.154022    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"]
Nov 02 12:40:28.648163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:28.648124    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" event={"ID":"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da","Type":"ContainerStarted","Data":"f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1"}
Nov 02 12:40:28.648353 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:28.648175    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" event={"ID":"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da","Type":"ContainerStarted","Data":"6b9736b78a743fb981364d7842f862dd2e167b121b98e211428e9f3fd29b462f"}
Nov 02 12:40:28.693012 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:28.692950    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podStartSLOduration=1.6929339140000002 podStartE2EDuration="1.692933914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:28.691831131 +0000 UTC m=+774.798533983" watchObservedRunningTime="2026-11-02 12:40:28.692933914 +0000 UTC m=+774.799636766"
Nov 02 12:40:28.842070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:28.842034    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:29.240332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:29.240274    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:30.777391 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:30.777352    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:32.363913 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:32.363880    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:33.947803 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:33.947764    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:34.226800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.226723    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:34.576127 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.576035    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"]
Nov 02 12:40:34.580299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.580279    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:34.599436 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.599413    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"]
Nov 02 12:40:34.609480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.609454    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:40:34.609645 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.609493    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:40:34.679175 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.679131    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e84a277-7f2vg\" (UID: \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:34.679345 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.679280    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e84a277-7f2vg\" (UID: \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:34.679453 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.679433    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv574\" (UniqueName: \"kubernetes.io/projected/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-kube-api-access-pv574\") pod \"maas-api-revoke-sub-9e84a277-7f2vg\" (UID: \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:34.780767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.780728    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv574\" (UniqueName: \"kubernetes.io/projected/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-kube-api-access-pv574\") pod \"maas-api-revoke-sub-9e84a277-7f2vg\" (UID: \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:34.780944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.780829    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e84a277-7f2vg\" (UID: \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:34.780944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.780914    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e84a277-7f2vg\" (UID: \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:34.781505 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.781479    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e84a277-7f2vg\" (UID: \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:34.783418 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.783396    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e84a277-7f2vg\" (UID: \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:34.805909 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.805886    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv574\" (UniqueName: \"kubernetes.io/projected/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-kube-api-access-pv574\") pod \"maas-api-revoke-sub-9e84a277-7f2vg\" (UID: \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:34.897075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:34.897043    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:35.212605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:35.212576    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"]
Nov 02 12:40:35.213041 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:40:35.212770    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab3d0ff_7fe2_4679_a45d_5aab696bdfff.slice/crio-5b3efc38b6c1454d7ceba5a51b5ce3601b3a87a3ce46cad563d2c9a12b269b73 WatchSource:0}: Error finding container 5b3efc38b6c1454d7ceba5a51b5ce3601b3a87a3ce46cad563d2c9a12b269b73: Status 404 returned error can't find the container with id 5b3efc38b6c1454d7ceba5a51b5ce3601b3a87a3ce46cad563d2c9a12b269b73
Nov 02 12:40:35.632289 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:35.632250    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:35.696908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:35.696875    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg" event={"ID":"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff","Type":"ContainerStarted","Data":"35d829d5b684a638006f359e9687504cc9385bf80029c964a4e584a970d4b4a8"}
Nov 02 12:40:35.697106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:35.696932    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg" event={"ID":"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff","Type":"ContainerStarted","Data":"5b3efc38b6c1454d7ceba5a51b5ce3601b3a87a3ce46cad563d2c9a12b269b73"}
Nov 02 12:40:35.740815 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:35.740764    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg" podStartSLOduration=1.74075242 podStartE2EDuration="1.74075242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:35.740494471 +0000 UTC m=+781.847197320" watchObservedRunningTime="2026-11-02 12:40:35.74075242 +0000 UTC m=+781.847455270"
Nov 02 12:40:36.413635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.413552    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"]
Nov 02 12:40:36.417765 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.417747    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:36.437855 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.437823    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"]
Nov 02 12:40:36.506328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.506277    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc90d70-wpvqs\" (UID: \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:36.506525 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.506463    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm477\" (UniqueName: \"kubernetes.io/projected/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-kube-api-access-wm477\") pod \"maas-api-revoke-sub-fdc90d70-wpvqs\" (UID: \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:36.506654 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.506629    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc90d70-wpvqs\" (UID: \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:36.607741 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.607706    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm477\" (UniqueName: \"kubernetes.io/projected/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-kube-api-access-wm477\") pod \"maas-api-revoke-sub-fdc90d70-wpvqs\" (UID: \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:36.607936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.607844    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc90d70-wpvqs\" (UID: \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:36.607936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.607937    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc90d70-wpvqs\" (UID: \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:36.608729 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.608704    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdc90d70-wpvqs\" (UID: \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:36.610799 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.610767    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdc90d70-wpvqs\" (UID: \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:36.632776 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.632749    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm477\" (UniqueName: \"kubernetes.io/projected/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-kube-api-access-wm477\") pod \"maas-api-revoke-sub-fdc90d70-wpvqs\" (UID: \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:36.705866 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.705779    5879 generic.go:356] "Generic (PLEG): container finished" podID="4ab3d0ff-7fe2-4679-a45d-5aab696bdfff" containerID="35d829d5b684a638006f359e9687504cc9385bf80029c964a4e584a970d4b4a8" exitCode=0
Nov 02 12:40:36.706044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.705868    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg" event={"ID":"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff","Type":"ContainerDied","Data":"35d829d5b684a638006f359e9687504cc9385bf80029c964a4e584a970d4b4a8"}
Nov 02 12:40:36.731460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:36.731428    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:37.040877 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:40:37.040845    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae2fc15_5e71_4c3a_89c6_dbf59fb58a08.slice/crio-6df55bb66d3f90949a1dd73760c40ebed196875858cc674d8b940e95a6f460ea WatchSource:0}: Error finding container 6df55bb66d3f90949a1dd73760c40ebed196875858cc674d8b940e95a6f460ea: Status 404 returned error can't find the container with id 6df55bb66d3f90949a1dd73760c40ebed196875858cc674d8b940e95a6f460ea
Nov 02 12:40:37.041154 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:37.041132    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"]
Nov 02 12:40:37.368192 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:37.368152    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:37.713843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:37.713806    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs" event={"ID":"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08","Type":"ContainerStarted","Data":"6d9e192135d11ef6057df305196c7fd3295285fba3b54e56567d37c101552977"}
Nov 02 12:40:37.714311 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:37.713871    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs" event={"ID":"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08","Type":"ContainerStarted","Data":"6df55bb66d3f90949a1dd73760c40ebed196875858cc674d8b940e95a6f460ea"}
Nov 02 12:40:37.757792 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:37.757739    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs" podStartSLOduration=1.75772599 podStartE2EDuration="1.75772599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:37.756606147 +0000 UTC m=+783.863308999" watchObservedRunningTime="2026-11-02 12:40:37.75772599 +0000 UTC m=+783.864428842"
Nov 02 12:40:37.845023 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:37.844387    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:37.885298 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:37.884833    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:37.924949 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:37.924921    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:38.034150 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.034107    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-xks-ca\") pod \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\" (UID: \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\") "
Nov 02 12:40:38.034312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.034193    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-openshift-ca\") pod \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\" (UID: \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\") "
Nov 02 12:40:38.034312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.034302    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-kube-api-access-pv574\" (UniqueName: \"kubernetes.io/projected/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-kube-api-access-pv574\") pod \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\" (UID: \"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff\") "
Nov 02 12:40:38.035815 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.035648    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-openshift-ca" pod "4ab3d0ff-7fe2-4679-a45d-5aab696bdfff" (UID: "4ab3d0ff-7fe2-4679-a45d-5aab696bdfff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:38.038318 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.037881    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-xks-ca" pod "4ab3d0ff-7fe2-4679-a45d-5aab696bdfff" (UID: "4ab3d0ff-7fe2-4679-a45d-5aab696bdfff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:40:38.038318 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.038122    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-kube-api-access-pv574" pod "4ab3d0ff-7fe2-4679-a45d-5aab696bdfff" (UID: "4ab3d0ff-7fe2-4679-a45d-5aab696bdfff"). InnerVolumeSpecName "kube-api-access-pv574". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:38.136455 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.136408    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:38.136931 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.136698    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:38.137115 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.136941    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv574\" (UniqueName: \"kubernetes.io/projected/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff-kube-api-access-pv574\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:38.679822 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.679710    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:38.767039 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.763723    5879 generic.go:356] "Generic (PLEG): container finished" podID="1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08" containerID="6d9e192135d11ef6057df305196c7fd3295285fba3b54e56567d37c101552977" exitCode=0
Nov 02 12:40:38.767039 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.763834    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs" event={"ID":"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08","Type":"ContainerDied","Data":"6d9e192135d11ef6057df305196c7fd3295285fba3b54e56567d37c101552977"}
Nov 02 12:40:38.767600 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.767511    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"
Nov 02 12:40:38.770022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.768480    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg" event={"ID":"4ab3d0ff-7fe2-4679-a45d-5aab696bdfff","Type":"ContainerDied","Data":"5b3efc38b6c1454d7ceba5a51b5ce3601b3a87a3ce46cad563d2c9a12b269b73"}
Nov 02 12:40:38.770022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.768537    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b3efc38b6c1454d7ceba5a51b5ce3601b3a87a3ce46cad563d2c9a12b269b73"
Nov 02 12:40:38.986039 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:38.985931    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:39.835624 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:39.835573    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:39.956370 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:39.956342    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:40.019490 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:40.019459    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-xks-ca\") pod \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\" (UID: \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\") "
Nov 02 12:40:40.019674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:40.019631    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-kube-api-access-wm477\" (UniqueName: \"kubernetes.io/projected/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-kube-api-access-wm477\") pod \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\" (UID: \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\") "
Nov 02 12:40:40.019761 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:40.019739    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-openshift-ca\") pod \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\" (UID: \"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08\") "
Nov 02 12:40:40.020220 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:40.020187    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-openshift-ca" pod "1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08" (UID: "1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:40.020524 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:40.020501    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:40.021830 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:40.021808    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-xks-ca" pod "1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08" (UID: "1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:40:40.022443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:40.022414    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-kube-api-access-wm477" pod "1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08" (UID: "1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08"). InnerVolumeSpecName "kube-api-access-wm477". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:40.121754 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:40.121712    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm477\" (UniqueName: \"kubernetes.io/projected/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-kube-api-access-wm477\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:40.121754 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:40.121748    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:40.783004 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:40.782969    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"
Nov 02 12:40:40.783170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:40.782982    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs" event={"ID":"1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08","Type":"ContainerDied","Data":"6df55bb66d3f90949a1dd73760c40ebed196875858cc674d8b940e95a6f460ea"}
Nov 02 12:40:40.783170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:40.783122    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6df55bb66d3f90949a1dd73760c40ebed196875858cc674d8b940e95a6f460ea"
Nov 02 12:40:41.454949 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:41.454918    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:42.366471 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.366440    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"]
Nov 02 12:40:42.366973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.366961    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4ab3d0ff-7fe2-4679-a45d-5aab696bdfff" containerName="revoke-keys"
Nov 02 12:40:42.367061 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.366976    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab3d0ff-7fe2-4679-a45d-5aab696bdfff" containerName="revoke-keys"
Nov 02 12:40:42.367061 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.367053    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08" containerName="revoke-keys"
Nov 02 12:40:42.367061 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.367060    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08" containerName="revoke-keys"
Nov 02 12:40:42.367190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.367179    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08" containerName="revoke-keys"
Nov 02 12:40:42.367225 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.367195    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab3d0ff-7fe2-4679-a45d-5aab696bdfff" containerName="revoke-keys"
Nov 02 12:40:42.371442 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.371421    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.403876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.403757    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"]
Nov 02 12:40:42.448343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.448299    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.448343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.448345    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.448598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.448480    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.448598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.448516    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlsgm\" (UniqueName: \"kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-kube-api-access-rlsgm\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.448679 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.448627    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.448734 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.448676    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.448793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.448768    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5115dd72-ef9a-4a46-b84a-ee44b699082c-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.448849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.448823    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.448962 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.448948    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.550258 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.550214    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.550781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.550268    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlsgm\" (UniqueName: \"kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-kube-api-access-rlsgm\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.550781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.550342    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.550781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.550382    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.550781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.550431    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5115dd72-ef9a-4a46-b84a-ee44b699082c-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.550781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.550484    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.550781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.550582    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.550781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.550638    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.550781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.550649    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.550781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.550672    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.551235 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.550910    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.551235 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.551223    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.551318 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.551236    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.551318 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.551292    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5115dd72-ef9a-4a46-b84a-ee44b699082c-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.552742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.552723    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.553278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.553256    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.575732 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.575699    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.575926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.575897    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlsgm\" (UniqueName: \"kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-kube-api-access-rlsgm\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:42.686059 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:42.685969    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:43.005466 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:40:43.005430    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5115dd72_ef9a_4a46_b84a_ee44b699082c.slice/crio-9c01eb1c7d1fbce8ab986fb95fa0657cb1e8a07d0ed1cafd0f54b1dac79458d2 WatchSource:0}: Error finding container 9c01eb1c7d1fbce8ab986fb95fa0657cb1e8a07d0ed1cafd0f54b1dac79458d2: Status 404 returned error can't find the container with id 9c01eb1c7d1fbce8ab986fb95fa0657cb1e8a07d0ed1cafd0f54b1dac79458d2
Nov 02 12:40:43.006411 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:43.006334    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"]
Nov 02 12:40:43.008470 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:43.008435    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:40:43.008550 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:43.008508    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:40:43.008550 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:43.008544    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:40:43.037766 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:43.037741    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:43.813483 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:43.813437    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" event={"ID":"5115dd72-ef9a-4a46-b84a-ee44b699082c","Type":"ContainerStarted","Data":"87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4"}
Nov 02 12:40:43.814040 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:43.813503    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" event={"ID":"5115dd72-ef9a-4a46-b84a-ee44b699082c","Type":"ContainerStarted","Data":"9c01eb1c7d1fbce8ab986fb95fa0657cb1e8a07d0ed1cafd0f54b1dac79458d2"}
Nov 02 12:40:43.878192 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:43.878134    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" podStartSLOduration=1.8781191449999999 podStartE2EDuration="1.878119145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:43.875363598 +0000 UTC m=+789.982066451" watchObservedRunningTime="2026-11-02 12:40:43.878119145 +0000 UTC m=+789.984821995"
Nov 02 12:40:44.701418 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:44.701342    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:44.703609 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:44.703582    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 12:40:44.703815 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:44.703790    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" podUID="5115dd72-ef9a-4a46-b84a-ee44b699082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 12:40:44.770814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:44.770786    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:44.852382 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:40:44.851817    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:40:44.914882 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:44.914850    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"]
Nov 02 12:40:44.919190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:44.919164    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:44.939939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:44.939908    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"]
Nov 02 12:40:44.948820 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:44.948790    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:40:44.949016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:44.948851    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:40:45.080386 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.080294    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e25d58-xpppq\" (UID: \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:45.080564 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.080418    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp5hz\" (UniqueName: \"kubernetes.io/projected/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-kube-api-access-kp5hz\") pod \"maas-api-revoke-sub-10e25d58-xpppq\" (UID: \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:45.080564 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.080503    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e25d58-xpppq\" (UID: \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:45.181723 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.181680    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e25d58-xpppq\" (UID: \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:45.190016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.182599    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e25d58-xpppq\" (UID: \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:45.190435 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.190406    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp5hz\" (UniqueName: \"kubernetes.io/projected/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-kube-api-access-kp5hz\") pod \"maas-api-revoke-sub-10e25d58-xpppq\" (UID: \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:45.206593 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.191758    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10e25d58-xpppq\" (UID: \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:45.208369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.208345    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10e25d58-xpppq\" (UID: \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:45.215358 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.215329    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp5hz\" (UniqueName: \"kubernetes.io/projected/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-kube-api-access-kp5hz\") pod \"maas-api-revoke-sub-10e25d58-xpppq\" (UID: \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:45.235321 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.235291    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:45.565750 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:40:45.565705    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67da26be_d09c_4c0d_99f1_b7f6a49dcc73.slice/crio-ce2c8ee6b05e28e8209b8910427ab4b3051ed2756ae9ac1752d6dfe087339620 WatchSource:0}: Error finding container ce2c8ee6b05e28e8209b8910427ab4b3051ed2756ae9ac1752d6dfe087339620: Status 404 returned error can't find the container with id ce2c8ee6b05e28e8209b8910427ab4b3051ed2756ae9ac1752d6dfe087339620
Nov 02 12:40:45.566687 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.566659    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"]
Nov 02 12:40:45.692022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.690824    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 12:40:45.692022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.690889    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" podUID="5115dd72-ef9a-4a46-b84a-ee44b699082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 12:40:45.832052 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:45.832016    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq" event={"ID":"67da26be-d09c-4c0d-99f1-b7f6a49dcc73","Type":"ContainerStarted","Data":"ce2c8ee6b05e28e8209b8910427ab4b3051ed2756ae9ac1752d6dfe087339620"}
Nov 02 12:40:46.440311 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:46.440199    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:46.697138 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:46.688159    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 12:40:46.697138 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:46.688218    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" podUID="5115dd72-ef9a-4a46-b84a-ee44b699082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 12:40:46.840099 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:46.840051    5879 generic.go:356] "Generic (PLEG): container finished" podID="67da26be-d09c-4c0d-99f1-b7f6a49dcc73" containerID="e9cb49b282427b3b7c2c455bed1bcb0fa2d4acad42332386f0406653f41208f5" exitCode=0
Nov 02 12:40:46.840311 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:46.840096    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq" event={"ID":"67da26be-d09c-4c0d-99f1-b7f6a49dcc73","Type":"ContainerDied","Data":"e9cb49b282427b3b7c2c455bed1bcb0fa2d4acad42332386f0406653f41208f5"}
Nov 02 12:40:46.935748 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:46.935711    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"]
Nov 02 12:40:46.941637 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:46.941599    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:46.959678 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:46.959648    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"]
Nov 02 12:40:47.131258 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.131217    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c3344ad-8x8bk\" (UID: \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:47.131471 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.131298    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c3344ad-8x8bk\" (UID: \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:47.131471 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.131419    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xxm\" (UniqueName: \"kubernetes.io/projected/7290a2ed-2d91-49af-a77e-bfb624c70ca7-kube-api-access-z8xxm\") pod \"maas-api-revoke-sub-5c3344ad-8x8bk\" (UID: \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:47.232353 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.232255    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c3344ad-8x8bk\" (UID: \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:47.232353 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.232315    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c3344ad-8x8bk\" (UID: \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:47.232543 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.232373    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xxm\" (UniqueName: \"kubernetes.io/projected/7290a2ed-2d91-49af-a77e-bfb624c70ca7-kube-api-access-z8xxm\") pod \"maas-api-revoke-sub-5c3344ad-8x8bk\" (UID: \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:47.233197 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.233169    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c3344ad-8x8bk\" (UID: \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:47.235614 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.235586    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c3344ad-8x8bk\" (UID: \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:47.263088 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.263053    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xxm\" (UniqueName: \"kubernetes.io/projected/7290a2ed-2d91-49af-a77e-bfb624c70ca7-kube-api-access-z8xxm\") pod \"maas-api-revoke-sub-5c3344ad-8x8bk\" (UID: \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:47.561055 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.560931    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:47.703018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.687421    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 12:40:47.703018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.687493    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" podUID="5115dd72-ef9a-4a46-b84a-ee44b699082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 12:40:47.920208 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:47.919696    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"]
Nov 02 12:40:47.923185 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:40:47.923152    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7290a2ed_2d91_49af_a77e_bfb624c70ca7.slice/crio-133c4d5ee21b9510907d2f0720fa7ec57381d62d328d3f15e5575214c561e41e WatchSource:0}: Error finding container 133c4d5ee21b9510907d2f0720fa7ec57381d62d328d3f15e5575214c561e41e: Status 404 returned error can't find the container with id 133c4d5ee21b9510907d2f0720fa7ec57381d62d328d3f15e5575214c561e41e
Nov 02 12:40:48.123116 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.123091    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:48.144136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.144104    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-kube-api-access-kp5hz\" (UniqueName: \"kubernetes.io/projected/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-kube-api-access-kp5hz\") pod \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\" (UID: \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\") "
Nov 02 12:40:48.144311 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.144210    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-openshift-ca\") pod \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\" (UID: \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\") "
Nov 02 12:40:48.144311 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.144304    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-xks-ca\") pod \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\" (UID: \"67da26be-d09c-4c0d-99f1-b7f6a49dcc73\") "
Nov 02 12:40:48.144832 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.144802    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-openshift-ca" pod "67da26be-d09c-4c0d-99f1-b7f6a49dcc73" (UID: "67da26be-d09c-4c0d-99f1-b7f6a49dcc73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:48.147027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.146957    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-xks-ca" pod "67da26be-d09c-4c0d-99f1-b7f6a49dcc73" (UID: "67da26be-d09c-4c0d-99f1-b7f6a49dcc73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:40:48.147167 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.147137    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-kube-api-access-kp5hz" pod "67da26be-d09c-4c0d-99f1-b7f6a49dcc73" (UID: "67da26be-d09c-4c0d-99f1-b7f6a49dcc73"). InnerVolumeSpecName "kube-api-access-kp5hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:48.229293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.229262    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:48.248463 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.248078    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp5hz\" (UniqueName: \"kubernetes.io/projected/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-kube-api-access-kp5hz\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:48.248463 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.248151    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:48.248463 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.248173    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67da26be-d09c-4c0d-99f1-b7f6a49dcc73-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:48.690022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.686676    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 12:40:48.690022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.686739    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" podUID="5115dd72-ef9a-4a46-b84a-ee44b699082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 12:40:48.814480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.814355    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"]
Nov 02 12:40:48.815245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.815227    5879 cpu_manager.go:414] "containerMap: removing container" podUID="67da26be-d09c-4c0d-99f1-b7f6a49dcc73" containerName="revoke-keys"
Nov 02 12:40:48.815315 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.815248    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67da26be-d09c-4c0d-99f1-b7f6a49dcc73" containerName="revoke-keys"
Nov 02 12:40:48.815443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.815431    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="67da26be-d09c-4c0d-99f1-b7f6a49dcc73" containerName="revoke-keys"
Nov 02 12:40:48.819616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.819588    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:48.833574 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.833522    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"]
Nov 02 12:40:48.855606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.855417    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f134cf3-vk5tg\" (UID: \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:48.855606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.855468    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f134cf3-vk5tg\" (UID: \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:48.855606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.855543    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9q44\" (UniqueName: \"kubernetes.io/projected/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-kube-api-access-j9q44\") pod \"maas-api-revoke-sub-8f134cf3-vk5tg\" (UID: \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:48.875125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.873964    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq" event={"ID":"67da26be-d09c-4c0d-99f1-b7f6a49dcc73","Type":"ContainerDied","Data":"ce2c8ee6b05e28e8209b8910427ab4b3051ed2756ae9ac1752d6dfe087339620"}
Nov 02 12:40:48.875125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.874035    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce2c8ee6b05e28e8209b8910427ab4b3051ed2756ae9ac1752d6dfe087339620"
Nov 02 12:40:48.875125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.874126    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"
Nov 02 12:40:48.876450 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.876417    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk" event={"ID":"7290a2ed-2d91-49af-a77e-bfb624c70ca7","Type":"ContainerStarted","Data":"249afb2059113c327b97bb2b2ae990e8cf13d5a5d74a4074a493df92a4012068"}
Nov 02 12:40:48.876662 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.876643    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk" event={"ID":"7290a2ed-2d91-49af-a77e-bfb624c70ca7","Type":"ContainerStarted","Data":"133c4d5ee21b9510907d2f0720fa7ec57381d62d328d3f15e5575214c561e41e"}
Nov 02 12:40:48.923466 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.923405    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk" podStartSLOduration=2.923385666 podStartE2EDuration="2.923385666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:48.920726127 +0000 UTC m=+795.027428979" watchObservedRunningTime="2026-11-02 12:40:48.923385666 +0000 UTC m=+795.030088517"
Nov 02 12:40:48.957391 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.957302    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f134cf3-vk5tg\" (UID: \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:48.957391 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.957351    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f134cf3-vk5tg\" (UID: \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:48.957599 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.957544    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9q44\" (UniqueName: \"kubernetes.io/projected/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-kube-api-access-j9q44\") pod \"maas-api-revoke-sub-8f134cf3-vk5tg\" (UID: \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:48.958168 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.958138    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f134cf3-vk5tg\" (UID: \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:48.960130 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:48.960105    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f134cf3-vk5tg\" (UID: \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:49.005625 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.005575    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9q44\" (UniqueName: \"kubernetes.io/projected/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-kube-api-access-j9q44\") pod \"maas-api-revoke-sub-8f134cf3-vk5tg\" (UID: \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:49.138514 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.138481    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:49.458908 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:40:49.458873    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbbfd85_88d0_4511_b66d_42d5c2fdfa75.slice/crio-0d7f61acf5b2d8ff9fd19d25077da322c13cd4cfb0c45ed338d459c8c4397253 WatchSource:0}: Error finding container 0d7f61acf5b2d8ff9fd19d25077da322c13cd4cfb0c45ed338d459c8c4397253: Status 404 returned error can't find the container with id 0d7f61acf5b2d8ff9fd19d25077da322c13cd4cfb0c45ed338d459c8c4397253
Nov 02 12:40:49.459981 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.459955    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"]
Nov 02 12:40:49.686528 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.686477    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 12:40:49.686717 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.686540    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" podUID="5115dd72-ef9a-4a46-b84a-ee44b699082c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 12:40:49.690779 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.690734    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"]
Nov 02 12:40:49.696430 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.696400    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.721287 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.721189    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-61e341-openshift-default-dockercfg-jcvsw\""
Nov 02 12:40:49.730735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.730639    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"]
Nov 02 12:40:49.770340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.770300    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.770504 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.770391    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.770561 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.770501    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.770634 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.770576    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.770634 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.770607    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.770634 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.770625    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960e55b4-4d31-4210-b756-28c039d88f7f-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.770761 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.770717    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960e55b4-4d31-4210-b756-28c039d88f7f-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.770881 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.770859    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrkj\" (UniqueName: \"kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-kube-api-access-dbrkj\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.770942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.770917    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.872384 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.872347    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.872384 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.872396    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.872680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.872438    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.872680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.872458    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960e55b4-4d31-4210-b756-28c039d88f7f-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.872680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.872584    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960e55b4-4d31-4210-b756-28c039d88f7f-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.872680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.872654    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrkj\" (UniqueName: \"kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-kube-api-access-dbrkj\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.872912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.872701    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.872912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.872795    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.872912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.872825    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.872912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.872824    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.872912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.872842    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.873222 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.872935    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.873222 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.873110    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.873348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.873306    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960e55b4-4d31-4210-b756-28c039d88f7f-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.882201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.882169    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.882938 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.882911    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960e55b4-4d31-4210-b756-28c039d88f7f-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.887285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.887258    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg" event={"ID":"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75","Type":"ContainerStarted","Data":"0d7f61acf5b2d8ff9fd19d25077da322c13cd4cfb0c45ed338d459c8c4397253"}
Nov 02 12:40:49.889436 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.889396    5879 generic.go:356] "Generic (PLEG): container finished" podID="7290a2ed-2d91-49af-a77e-bfb624c70ca7" containerID="249afb2059113c327b97bb2b2ae990e8cf13d5a5d74a4074a493df92a4012068" exitCode=0
Nov 02 12:40:49.889598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.889485    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk" event={"ID":"7290a2ed-2d91-49af-a77e-bfb624c70ca7","Type":"ContainerDied","Data":"249afb2059113c327b97bb2b2ae990e8cf13d5a5d74a4074a493df92a4012068"}
Nov 02 12:40:49.899419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.899164    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrkj\" (UniqueName: \"kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-kube-api-access-dbrkj\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.899419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.899250    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:49.959889 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:49.959822    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:50.011505 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.011146    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:50.357640 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:40:50.357604    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960e55b4_4d31_4210_b756_28c039d88f7f.slice/crio-a9c1e7fbaaa240065fa5eb1c5d6883fc06cb3a2a147bd1939af744de12ec5c51 WatchSource:0}: Error finding container a9c1e7fbaaa240065fa5eb1c5d6883fc06cb3a2a147bd1939af744de12ec5c51: Status 404 returned error can't find the container with id a9c1e7fbaaa240065fa5eb1c5d6883fc06cb3a2a147bd1939af744de12ec5c51
Nov 02 12:40:50.358499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.358389    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"]
Nov 02 12:40:50.360733 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.360699    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:40:50.360848 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.360808    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:40:50.360913 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.360850    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:40:50.692234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.692202    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:50.692768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.692659    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:50.693538 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.693515    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 12:40:50.898761 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.898726    5879 generic.go:356] "Generic (PLEG): container finished" podID="5cbbfd85-88d0-4511-b66d-42d5c2fdfa75" containerID="2d11e6f0d2481ab95056a80111213f68be571a23bd56acb29c22cf1fe66d3600" exitCode=0
Nov 02 12:40:50.898931 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.898818    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg" event={"ID":"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75","Type":"ContainerDied","Data":"2d11e6f0d2481ab95056a80111213f68be571a23bd56acb29c22cf1fe66d3600"}
Nov 02 12:40:50.900613 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.900580    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" event={"ID":"960e55b4-4d31-4210-b756-28c039d88f7f","Type":"ContainerStarted","Data":"a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983"}
Nov 02 12:40:50.900768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.900628    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" event={"ID":"960e55b4-4d31-4210-b756-28c039d88f7f","Type":"ContainerStarted","Data":"a9c1e7fbaaa240065fa5eb1c5d6883fc06cb3a2a147bd1939af744de12ec5c51"}
Nov 02 12:40:50.995263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:50.995204    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podStartSLOduration=1.995184097 podStartE2EDuration="1.995184097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:40:50.994154822 +0000 UTC m=+797.100857676" watchObservedRunningTime="2026-11-02 12:40:50.995184097 +0000 UTC m=+797.101886953"
Nov 02 12:40:51.032771 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.029087    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:40:51.036866 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.036826    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:40:51.037122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.036894    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 12:40:51.069677 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.069650    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:51.095025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.094610    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-openshift-ca\") pod \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\" (UID: \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\") "
Nov 02 12:40:51.095025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.094828    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7290a2ed-2d91-49af-a77e-bfb624c70ca7-kube-api-access-z8xxm\" (UniqueName: \"kubernetes.io/projected/7290a2ed-2d91-49af-a77e-bfb624c70ca7-kube-api-access-z8xxm\") pod \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\" (UID: \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\") "
Nov 02 12:40:51.095025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.094966    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-xks-ca\") pod \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\" (UID: \"7290a2ed-2d91-49af-a77e-bfb624c70ca7\") "
Nov 02 12:40:51.112020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.096608    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-openshift-ca" pod "7290a2ed-2d91-49af-a77e-bfb624c70ca7" (UID: "7290a2ed-2d91-49af-a77e-bfb624c70ca7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:51.113477 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.113437    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-xks-ca" pod "7290a2ed-2d91-49af-a77e-bfb624c70ca7" (UID: "7290a2ed-2d91-49af-a77e-bfb624c70ca7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:40:51.114300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.114272    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7290a2ed-2d91-49af-a77e-bfb624c70ca7-kube-api-access-z8xxm" pod "7290a2ed-2d91-49af-a77e-bfb624c70ca7" (UID: "7290a2ed-2d91-49af-a77e-bfb624c70ca7"). InnerVolumeSpecName "kube-api-access-z8xxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:51.205018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.198069    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8xxm\" (UniqueName: \"kubernetes.io/projected/7290a2ed-2d91-49af-a77e-bfb624c70ca7-kube-api-access-z8xxm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:51.205018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.198116    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:51.205018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.198134    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7290a2ed-2d91-49af-a77e-bfb624c70ca7-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:51.702020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.696803    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:51.913803 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.913772    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"
Nov 02 12:40:51.913803 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.913796    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk" event={"ID":"7290a2ed-2d91-49af-a77e-bfb624c70ca7","Type":"ContainerDied","Data":"133c4d5ee21b9510907d2f0720fa7ec57381d62d328d3f15e5575214c561e41e"}
Nov 02 12:40:51.914117 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:51.913846    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="133c4d5ee21b9510907d2f0720fa7ec57381d62d328d3f15e5575214c561e41e"
Nov 02 12:40:52.012592 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.012552    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:40:52.012763 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.012617    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 12:40:52.117737 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.117708    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:52.216897 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.216849    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-openshift-ca\") pod \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\" (UID: \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\") "
Nov 02 12:40:52.216897 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.216905    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-kube-api-access-j9q44\" (UniqueName: \"kubernetes.io/projected/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-kube-api-access-j9q44\") pod \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\" (UID: \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\") "
Nov 02 12:40:52.217176 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.217155    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-xks-ca\") pod \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\" (UID: \"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75\") "
Nov 02 12:40:52.217354 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.217325    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-openshift-ca" pod "5cbbfd85-88d0-4511-b66d-42d5c2fdfa75" (UID: "5cbbfd85-88d0-4511-b66d-42d5c2fdfa75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:40:52.217651 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.217630    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:52.219979 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.219891    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-xks-ca" pod "5cbbfd85-88d0-4511-b66d-42d5c2fdfa75" (UID: "5cbbfd85-88d0-4511-b66d-42d5c2fdfa75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:40:52.220237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.220198    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-kube-api-access-j9q44" pod "5cbbfd85-88d0-4511-b66d-42d5c2fdfa75" (UID: "5cbbfd85-88d0-4511-b66d-42d5c2fdfa75"). InnerVolumeSpecName "kube-api-access-j9q44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:40:52.318513 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.318470    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:52.318513 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.318503    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9q44\" (UniqueName: \"kubernetes.io/projected/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75-kube-api-access-j9q44\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:40:52.921774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.921742    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"
Nov 02 12:40:52.921774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.921749    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg" event={"ID":"5cbbfd85-88d0-4511-b66d-42d5c2fdfa75","Type":"ContainerDied","Data":"0d7f61acf5b2d8ff9fd19d25077da322c13cd4cfb0c45ed338d459c8c4397253"}
Nov 02 12:40:52.922343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:52.921829    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d7f61acf5b2d8ff9fd19d25077da322c13cd4cfb0c45ed338d459c8c4397253"
Nov 02 12:40:53.012511 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:53.012470    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:40:53.012688 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:53.012520    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 12:40:53.442395 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:53.442359    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:54.014653 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:54.014611    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:40:54.015167 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:54.014677    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 12:40:55.012196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:55.012157    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:40:55.012416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:55.012217    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 12:40:55.152406 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:55.152367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:40:56.029021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:56.028026    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:40:56.029021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:56.028117    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 12:40:56.777152 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:56.777106    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.012419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:57.012381    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:40:57.012704 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:57.012677    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 12:40:57.084676 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:57.083477    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.805396 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:57.805340    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.806608 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:57.806585    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:57.808310 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:57.808284    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 12:40:57.903835 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:57.903785    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:57.957020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:57.953127    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 12:40:58.119078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:58.115671    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:58.119078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:58.116815    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:58.145020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:58.120782    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 02 12:40:58.320021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:58.319123    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 02 12:40:58.320021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:58.145635    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:40:58.342654 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:58.341681    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:40:58.402102 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:58.399085    5879 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Nov 02 12:40:58.624841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:58.624809    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:40:58.680970 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:58.680223    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:40:59.046500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:40:59.046395    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:00.060197 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:00.060162    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:41:00.060617 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:00.060237    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:00.697307 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:00.697263    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:41:00.697728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:00.697707    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:41:00.698810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:00.698791    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 12:41:00.746810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:00.746774    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:00.910271 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:00.910239    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:01.080372 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:01.080239    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:02.006209 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.006175    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"]
Nov 02 12:41:02.006753 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.006739    5879 cpu_manager.go:414] "containerMap: removing container" podUID="7290a2ed-2d91-49af-a77e-bfb624c70ca7" containerName="revoke-keys"
Nov 02 12:41:02.006798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.006756    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7290a2ed-2d91-49af-a77e-bfb624c70ca7" containerName="revoke-keys"
Nov 02 12:41:02.006798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.006766    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5cbbfd85-88d0-4511-b66d-42d5c2fdfa75" containerName="revoke-keys"
Nov 02 12:41:02.006798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.006771    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbbfd85-88d0-4511-b66d-42d5c2fdfa75" containerName="revoke-keys"
Nov 02 12:41:02.006880 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.006871    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="7290a2ed-2d91-49af-a77e-bfb624c70ca7" containerName="revoke-keys"
Nov 02 12:41:02.006880 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.006879    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbbfd85-88d0-4511-b66d-42d5c2fdfa75" containerName="revoke-keys"
Nov 02 12:41:02.009642 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.009619    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:02.011590 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.011574    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:02.028410 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.028382    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:41:02.030616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.030590    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"]
Nov 02 12:41:02.040653 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.040264    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:41:02.054010 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.053958    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqz4f\" (UniqueName: \"kubernetes.io/projected/a4015f4c-88fe-48a4-a19a-918cb1333bfe-kube-api-access-wqz4f\") pod \"maas-api-revoke-sub-eaa85885-c86hr\" (UID: \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:02.054193 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.054168    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaa85885-c86hr\" (UID: \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:02.054311 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.054293    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaa85885-c86hr\" (UID: \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:02.144841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.144803    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:02.155468 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.155440    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqz4f\" (UniqueName: \"kubernetes.io/projected/a4015f4c-88fe-48a4-a19a-918cb1333bfe-kube-api-access-wqz4f\") pod \"maas-api-revoke-sub-eaa85885-c86hr\" (UID: \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:02.155614 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.155570    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaa85885-c86hr\" (UID: \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:02.155703 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.155688    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaa85885-c86hr\" (UID: \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:02.156301 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.156272    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaa85885-c86hr\" (UID: \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:02.158240 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.158222    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaa85885-c86hr\" (UID: \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:02.180954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.180923    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqz4f\" (UniqueName: \"kubernetes.io/projected/a4015f4c-88fe-48a4-a19a-918cb1333bfe-kube-api-access-wqz4f\") pod \"maas-api-revoke-sub-eaa85885-c86hr\" (UID: \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:02.326452 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.326367    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:02.638036 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:41:02.635599    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4015f4c_88fe_48a4_a19a_918cb1333bfe.slice/crio-6b50f59aac304ad278d4695a9b68b60315d78d43656e824a0ace06f3ced996a7 WatchSource:0}: Error finding container 6b50f59aac304ad278d4695a9b68b60315d78d43656e824a0ace06f3ced996a7: Status 404 returned error can't find the container with id 6b50f59aac304ad278d4695a9b68b60315d78d43656e824a0ace06f3ced996a7
Nov 02 12:41:02.638036 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.636584    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"]
Nov 02 12:41:02.751392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:02.751353    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:03.365817 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.365774    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr" event={"ID":"a4015f4c-88fe-48a4-a19a-918cb1333bfe","Type":"ContainerStarted","Data":"1737e911df9ad778822c18f7d6cfb388c1c6530c43a76515f1033f8be8db3b11"}
Nov 02 12:41:03.366256 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.365840    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr" event={"ID":"a4015f4c-88fe-48a4-a19a-918cb1333bfe","Type":"ContainerStarted","Data":"6b50f59aac304ad278d4695a9b68b60315d78d43656e824a0ace06f3ced996a7"}
Nov 02 12:41:03.412768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.412707    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr" podStartSLOduration=2.412691514 podStartE2EDuration="2.412691514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:03.411672028 +0000 UTC m=+809.518374880" watchObservedRunningTime="2026-11-02 12:41:03.412691514 +0000 UTC m=+809.519394364"
Nov 02 12:41:03.599329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.599300    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"]
Nov 02 12:41:03.603360 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.603343    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:03.623343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.623267    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"]
Nov 02 12:41:03.674947 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.674914    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0a4fb5b-c7tdj\" (UID: \"84c68a59-59f8-47a4-a210-db61dacfda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:03.675163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.675076    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0a4fb5b-c7tdj\" (UID: \"84c68a59-59f8-47a4-a210-db61dacfda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:03.675288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.675268    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k7nl\" (UniqueName: \"kubernetes.io/projected/84c68a59-59f8-47a4-a210-db61dacfda90-kube-api-access-7k7nl\") pod \"maas-api-revoke-sub-a0a4fb5b-c7tdj\" (UID: \"84c68a59-59f8-47a4-a210-db61dacfda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:03.776794 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.776763    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0a4fb5b-c7tdj\" (UID: \"84c68a59-59f8-47a4-a210-db61dacfda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:03.776962 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.776902    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k7nl\" (UniqueName: \"kubernetes.io/projected/84c68a59-59f8-47a4-a210-db61dacfda90-kube-api-access-7k7nl\") pod \"maas-api-revoke-sub-a0a4fb5b-c7tdj\" (UID: \"84c68a59-59f8-47a4-a210-db61dacfda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:03.776962 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.776953    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0a4fb5b-c7tdj\" (UID: \"84c68a59-59f8-47a4-a210-db61dacfda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:03.777535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.777508    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0a4fb5b-c7tdj\" (UID: \"84c68a59-59f8-47a4-a210-db61dacfda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:03.779602 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.779578    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0a4fb5b-c7tdj\" (UID: \"84c68a59-59f8-47a4-a210-db61dacfda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:03.802520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.802492    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k7nl\" (UniqueName: \"kubernetes.io/projected/84c68a59-59f8-47a4-a210-db61dacfda90-kube-api-access-7k7nl\") pod \"maas-api-revoke-sub-a0a4fb5b-c7tdj\" (UID: \"84c68a59-59f8-47a4-a210-db61dacfda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:03.960505 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:03.960425    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:04.267889 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:41:04.267850    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c68a59_59f8_47a4_a210_db61dacfda90.slice/crio-020543755a9c6770a68ae0db8b8b46a268cf4852a7ab1481dc0a3ec1a12f0bfa WatchSource:0}: Error finding container 020543755a9c6770a68ae0db8b8b46a268cf4852a7ab1481dc0a3ec1a12f0bfa: Status 404 returned error can't find the container with id 020543755a9c6770a68ae0db8b8b46a268cf4852a7ab1481dc0a3ec1a12f0bfa
Nov 02 12:41:04.268505 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:04.268483    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"]
Nov 02 12:41:04.374959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:04.374930    5879 generic.go:356] "Generic (PLEG): container finished" podID="a4015f4c-88fe-48a4-a19a-918cb1333bfe" containerID="1737e911df9ad778822c18f7d6cfb388c1c6530c43a76515f1033f8be8db3b11" exitCode=0
Nov 02 12:41:04.375416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:04.375045    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr" event={"ID":"a4015f4c-88fe-48a4-a19a-918cb1333bfe","Type":"ContainerDied","Data":"1737e911df9ad778822c18f7d6cfb388c1c6530c43a76515f1033f8be8db3b11"}
Nov 02 12:41:04.375935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:04.375885    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:04.376897 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:04.376871    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj" event={"ID":"84c68a59-59f8-47a4-a210-db61dacfda90","Type":"ContainerStarted","Data":"020543755a9c6770a68ae0db8b8b46a268cf4852a7ab1481dc0a3ec1a12f0bfa"}
Nov 02 12:41:05.385686 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:05.385652    5879 generic.go:356] "Generic (PLEG): container finished" podID="84c68a59-59f8-47a4-a210-db61dacfda90" containerID="3b2e8c047c071a9abdfe424064a6779514629ad384a1f893e903e04dc0b28c21" exitCode=0
Nov 02 12:41:05.386101 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:05.385741    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj" event={"ID":"84c68a59-59f8-47a4-a210-db61dacfda90","Type":"ContainerDied","Data":"3b2e8c047c071a9abdfe424064a6779514629ad384a1f893e903e04dc0b28c21"}
Nov 02 12:41:05.551811 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:05.551777    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:05.599602 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:05.599562    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-xks-ca\") pod \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\" (UID: \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\") "
Nov 02 12:41:05.599794 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:05.599637    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-openshift-ca\") pod \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\" (UID: \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\") "
Nov 02 12:41:05.599859 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:05.599826    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4015f4c-88fe-48a4-a19a-918cb1333bfe-kube-api-access-wqz4f\" (UniqueName: \"kubernetes.io/projected/a4015f4c-88fe-48a4-a19a-918cb1333bfe-kube-api-access-wqz4f\") pod \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\" (UID: \"a4015f4c-88fe-48a4-a19a-918cb1333bfe\") "
Nov 02 12:41:05.600882 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:05.600854    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-openshift-ca" pod "a4015f4c-88fe-48a4-a19a-918cb1333bfe" (UID: "a4015f4c-88fe-48a4-a19a-918cb1333bfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:41:05.605031 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:05.602366    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-xks-ca" pod "a4015f4c-88fe-48a4-a19a-918cb1333bfe" (UID: "a4015f4c-88fe-48a4-a19a-918cb1333bfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:41:05.605031 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:05.603471    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4015f4c-88fe-48a4-a19a-918cb1333bfe-kube-api-access-wqz4f" pod "a4015f4c-88fe-48a4-a19a-918cb1333bfe" (UID: "a4015f4c-88fe-48a4-a19a-918cb1333bfe"). InnerVolumeSpecName "kube-api-access-wqz4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:41:05.701545 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:05.701507    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqz4f\" (UniqueName: \"kubernetes.io/projected/a4015f4c-88fe-48a4-a19a-918cb1333bfe-kube-api-access-wqz4f\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:41:05.701545 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:05.701539    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:41:05.701545 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:05.701552    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4015f4c-88fe-48a4-a19a-918cb1333bfe-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:41:06.058710 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.058678    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:06.394389 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.394350    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"
Nov 02 12:41:06.394831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.394389    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr" event={"ID":"a4015f4c-88fe-48a4-a19a-918cb1333bfe","Type":"ContainerDied","Data":"6b50f59aac304ad278d4695a9b68b60315d78d43656e824a0ace06f3ced996a7"}
Nov 02 12:41:06.394831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.394444    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b50f59aac304ad278d4695a9b68b60315d78d43656e824a0ace06f3ced996a7"
Nov 02 12:41:06.578837 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.578810    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:06.612904 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.612875    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c68a59-59f8-47a4-a210-db61dacfda90-kube-api-access-7k7nl\" (UniqueName: \"kubernetes.io/projected/84c68a59-59f8-47a4-a210-db61dacfda90-kube-api-access-7k7nl\") pod \"84c68a59-59f8-47a4-a210-db61dacfda90\" (UID: \"84c68a59-59f8-47a4-a210-db61dacfda90\") "
Nov 02 12:41:06.613121 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.612926    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-openshift-ca\") pod \"84c68a59-59f8-47a4-a210-db61dacfda90\" (UID: \"84c68a59-59f8-47a4-a210-db61dacfda90\") "
Nov 02 12:41:06.613121 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.612974    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-xks-ca\") pod \"84c68a59-59f8-47a4-a210-db61dacfda90\" (UID: \"84c68a59-59f8-47a4-a210-db61dacfda90\") "
Nov 02 12:41:06.613365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.613342    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-openshift-ca" pod "84c68a59-59f8-47a4-a210-db61dacfda90" (UID: "84c68a59-59f8-47a4-a210-db61dacfda90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:41:06.613516 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.613497    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:41:06.615706 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.615678    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-xks-ca" pod "84c68a59-59f8-47a4-a210-db61dacfda90" (UID: "84c68a59-59f8-47a4-a210-db61dacfda90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:41:06.615706 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.615697    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c68a59-59f8-47a4-a210-db61dacfda90-kube-api-access-7k7nl" pod "84c68a59-59f8-47a4-a210-db61dacfda90" (UID: "84c68a59-59f8-47a4-a210-db61dacfda90"). InnerVolumeSpecName "kube-api-access-7k7nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:41:06.714320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.714287    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k7nl\" (UniqueName: \"kubernetes.io/projected/84c68a59-59f8-47a4-a210-db61dacfda90-kube-api-access-7k7nl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:41:06.714320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:06.714316    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c68a59-59f8-47a4-a210-db61dacfda90-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:41:07.402821 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:07.402792    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"
Nov 02 12:41:07.402821 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:07.402799    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj" event={"ID":"84c68a59-59f8-47a4-a210-db61dacfda90","Type":"ContainerDied","Data":"020543755a9c6770a68ae0db8b8b46a268cf4852a7ab1481dc0a3ec1a12f0bfa"}
Nov 02 12:41:07.403412 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:07.402882    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="020543755a9c6770a68ae0db8b8b46a268cf4852a7ab1481dc0a3ec1a12f0bfa"
Nov 02 12:41:07.780808 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:07.780693    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:09.398227 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:09.398191    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:11.047790 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:11.047755    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:11.191781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:11.191736    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:12.387462 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.387422    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"]
Nov 02 12:41:12.388277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.388245    5879 cpu_manager.go:414] "containerMap: removing container" podUID="84c68a59-59f8-47a4-a210-db61dacfda90" containerName="revoke-keys"
Nov 02 12:41:12.388277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.388271    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c68a59-59f8-47a4-a210-db61dacfda90" containerName="revoke-keys"
Nov 02 12:41:12.388558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.388315    5879 cpu_manager.go:414] "containerMap: removing container" podUID="a4015f4c-88fe-48a4-a19a-918cb1333bfe" containerName="revoke-keys"
Nov 02 12:41:12.388558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.388323    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4015f4c-88fe-48a4-a19a-918cb1333bfe" containerName="revoke-keys"
Nov 02 12:41:12.388558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.388456    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c68a59-59f8-47a4-a210-db61dacfda90" containerName="revoke-keys"
Nov 02 12:41:12.388558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.388472    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4015f4c-88fe-48a4-a19a-918cb1333bfe" containerName="revoke-keys"
Nov 02 12:41:12.393646 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.393626    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.422405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.422368    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"]
Nov 02 12:41:12.427563 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.427538    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:12.432218 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.432190    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"]
Nov 02 12:41:12.444819 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.444793    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:41:12.444975 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.444901    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:41:12.458038 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.453408    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"]
Nov 02 12:41:12.491849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.491783    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.491849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.491845    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.492170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.491953    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.492170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.491981    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2386118d-tmww6\" (UID: \"1ff2474e-cdfe-45f2-9326-28084a6aed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:12.492170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.492026    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.492170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.492067    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.492170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.492109    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsphf\" (UniqueName: \"kubernetes.io/projected/1ff2474e-cdfe-45f2-9326-28084a6aed91-kube-api-access-gsphf\") pod \"maas-api-revoke-sub-2386118d-tmww6\" (UID: \"1ff2474e-cdfe-45f2-9326-28084a6aed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:12.492170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.492142    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2386118d-tmww6\" (UID: \"1ff2474e-cdfe-45f2-9326-28084a6aed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:12.492170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.492171    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.492548 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.492203    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.492548 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.492247    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.492548 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.492271    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7hs\" (UniqueName: \"kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-kube-api-access-jx7hs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.593268 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593212    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.593268 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593264    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7hs\" (UniqueName: \"kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-kube-api-access-jx7hs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.593542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593386    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.593542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593430    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.593542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593516    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.593703 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593550    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2386118d-tmww6\" (UID: \"1ff2474e-cdfe-45f2-9326-28084a6aed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:12.593703 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593599    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.593703 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593667    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.593858 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593729    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsphf\" (UniqueName: \"kubernetes.io/projected/1ff2474e-cdfe-45f2-9326-28084a6aed91-kube-api-access-gsphf\") pod \"maas-api-revoke-sub-2386118d-tmww6\" (UID: \"1ff2474e-cdfe-45f2-9326-28084a6aed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:12.593858 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593752    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.593858 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593665    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.593858 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593790    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2386118d-tmww6\" (UID: \"1ff2474e-cdfe-45f2-9326-28084a6aed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:12.593858 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593838    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.594369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593890    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.594369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.593894    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.594369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.594030    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.594369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.594353    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.594753 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.594716    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2386118d-tmww6\" (UID: \"1ff2474e-cdfe-45f2-9326-28084a6aed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:12.596476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.596450    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2386118d-tmww6\" (UID: \"1ff2474e-cdfe-45f2-9326-28084a6aed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:12.596616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.596602    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.596682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.596615    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.618426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.618391    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.618632 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.618613    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7hs\" (UniqueName: \"kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-kube-api-access-jx7hs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.623380 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.623357    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsphf\" (UniqueName: \"kubernetes.io/projected/1ff2474e-cdfe-45f2-9326-28084a6aed91-kube-api-access-gsphf\") pod \"maas-api-revoke-sub-2386118d-tmww6\" (UID: \"1ff2474e-cdfe-45f2-9326-28084a6aed91\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:12.710035 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.709913    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:12.754258 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.754224    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:12.831385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:12.831333    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:13.045098 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.045057    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"]
Nov 02 12:41:13.047034 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:41:13.046924    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2844d7db_1a7a_4ebd_93c8_c49944bbb954.slice/crio-ca88e7fbff64d0b516a5e4bc0a732659016a92557cfe8087c8c66a7071f33a20 WatchSource:0}: Error finding container ca88e7fbff64d0b516a5e4bc0a732659016a92557cfe8087c8c66a7071f33a20: Status 404 returned error can't find the container with id ca88e7fbff64d0b516a5e4bc0a732659016a92557cfe8087c8c66a7071f33a20
Nov 02 12:41:13.050055 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.049949    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:13.050055 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.050058    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:13.050313 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.050098    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:13.076170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.076137    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"]
Nov 02 12:41:13.103365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.102088    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.118085 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:41:13.115846    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ff2474e_cdfe_45f2_9326_28084a6aed91.slice/crio-4775d4a6952bcd97abf46cee8b03ce8e85166eba7015df7e5305f7191b158372 WatchSource:0}: Error finding container 4775d4a6952bcd97abf46cee8b03ce8e85166eba7015df7e5305f7191b158372: Status 404 returned error can't find the container with id 4775d4a6952bcd97abf46cee8b03ce8e85166eba7015df7e5305f7191b158372
Nov 02 12:41:13.144615 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.144583    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"]
Nov 02 12:41:13.159475 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.159388    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"]
Nov 02 12:41:13.217804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.212940    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.218172 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.218138    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.219285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.219254    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cxcb\" (UniqueName: \"kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-kube-api-access-7cxcb\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.219561 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.219540    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.219733 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.219716    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.219931 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.219914    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea448abc-d63f-463c-9bae-67670be27f32-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.220163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.220143    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.221545 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.221521    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea448abc-d63f-463c-9bae-67670be27f32-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.221697 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.221680    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.325373    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea448abc-d63f-463c-9bae-67670be27f32-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.325427    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.325525    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.325579    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.325624    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxcb\" (UniqueName: \"kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-kube-api-access-7cxcb\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.325706    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.325756    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.325854    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea448abc-d63f-463c-9bae-67670be27f32-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.325924    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.329518    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.333883    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.334243    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.335916    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.340029    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea448abc-d63f-463c-9bae-67670be27f32-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.352021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.350645    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.367446 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.367378    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.379921 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.379854    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea448abc-d63f-463c-9bae-67670be27f32-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.443160 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.416124    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxcb\" (UniqueName: \"kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-kube-api-access-7cxcb\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.459006 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.458947    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" event={"ID":"2844d7db-1a7a-4ebd-93c8-c49944bbb954","Type":"ContainerStarted","Data":"493c25b10bb68b4dbdc8952ea40b910eb30cb7685c2c865f7fe37c4ec705c568"}
Nov 02 12:41:13.459216 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.459034    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" event={"ID":"2844d7db-1a7a-4ebd-93c8-c49944bbb954","Type":"ContainerStarted","Data":"ca88e7fbff64d0b516a5e4bc0a732659016a92557cfe8087c8c66a7071f33a20"}
Nov 02 12:41:13.461960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.461927    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6" event={"ID":"1ff2474e-cdfe-45f2-9326-28084a6aed91","Type":"ContainerStarted","Data":"4775d4a6952bcd97abf46cee8b03ce8e85166eba7015df7e5305f7191b158372"}
Nov 02 12:41:13.484319 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.481052    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:13.523661 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.523587    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podStartSLOduration=1.5235666829999999 podStartE2EDuration="1.523566683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:13.518316382 +0000 UTC m=+819.625019235" watchObservedRunningTime="2026-11-02 12:41:13.523566683 +0000 UTC m=+819.630269527"
Nov 02 12:41:13.715540 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:13.715469    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:14.283020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:14.282572    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:14.371245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:14.348071    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"]
Nov 02 12:41:14.478697 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:14.477132    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:14.478697 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:14.477222    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:14.478697 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:14.477280    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:14.585472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:14.585371    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" event={"ID":"ea448abc-d63f-463c-9bae-67670be27f32","Type":"ContainerStarted","Data":"9a91292773908949d0da4102ced12255dec9cf889c728a054bf16d1060950f41"}
Nov 02 12:41:14.621261 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:14.619699    5879 generic.go:356] "Generic (PLEG): container finished" podID="1ff2474e-cdfe-45f2-9326-28084a6aed91" containerID="e283f12e0ba2985dcc1ff8c4e5f5313c75f26fcabf0fcb735fe22c9ae7435e26" exitCode=0
Nov 02 12:41:14.621261 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:14.621175    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6" event={"ID":"1ff2474e-cdfe-45f2-9326-28084a6aed91","Type":"ContainerDied","Data":"e283f12e0ba2985dcc1ff8c4e5f5313c75f26fcabf0fcb735fe22c9ae7435e26"}
Nov 02 12:41:14.716160 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:14.716129    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:41:14.716390 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:14.716364    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 12:41:14.844076 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:14.841074    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.183063 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.180088    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.202052 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:41:15.189705    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:41:15.202052 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.200075    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.643145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.643087    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.705025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.704971    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" event={"ID":"ea448abc-d63f-463c-9bae-67670be27f32","Type":"ContainerStarted","Data":"74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d"}
Nov 02 12:41:15.726026 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.705299    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.726026 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.724743    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:41:15.726026 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.725342    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 12:41:15.792022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.790088    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podStartSLOduration=2.7900679630000003 podStartE2EDuration="2.790067963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:15.772344778 +0000 UTC m=+821.879047644" watchObservedRunningTime="2026-11-02 12:41:15.790067963 +0000 UTC m=+821.896770814"
Nov 02 12:41:15.819377 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.798847    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.859038 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.849090    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 12:41:15.859038 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.849809    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:41:15.878022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.875536    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:15.941076 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:15.940118    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.213220 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.213109    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.456021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.333879    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.585108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.537761    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 12:41:16.585108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.537821    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 12:41:16.585108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.538775    5879 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.681023 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.680963    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:16.688283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.688127    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.767534 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.761630    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.846343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.789605    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:41:16.846343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.789670    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 12:41:16.846343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.839739    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:16.846343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.839881    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:41:16.846343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.839940    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:41:16.846343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.840006    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:41:16.846343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.840045    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:41:16.895726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.855130    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.895726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.870247    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.895726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.872348    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:41:16.895726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.885590    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:16.895726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.885887    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:41:17.018080 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.948502    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.018080 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:16.980123    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:41:17.357085 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.159431    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:41:17.357085 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.159488    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.357085 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.243160    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.374961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.372335    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 02 12:41:17.534347 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.516835    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 12:41:17.534347 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.516897    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 12:41:17.719854 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.719816    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:41:17.720472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.720440    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 12:41:17.776305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.776270    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:17.777301 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.777266    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.849328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.804174    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-fx2tt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:41:17.849600 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.849567    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.849762 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.821869    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-xks-ca\") pod \"1ff2474e-cdfe-45f2-9326-28084a6aed91\" (UID: \"1ff2474e-cdfe-45f2-9326-28084a6aed91\") "
Nov 02 12:41:17.850090 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.850068    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff2474e-cdfe-45f2-9326-28084a6aed91-kube-api-access-gsphf\" (UniqueName: \"kubernetes.io/projected/1ff2474e-cdfe-45f2-9326-28084a6aed91-kube-api-access-gsphf\") pod \"1ff2474e-cdfe-45f2-9326-28084a6aed91\" (UID: \"1ff2474e-cdfe-45f2-9326-28084a6aed91\") "
Nov 02 12:41:17.852426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.852400    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-openshift-ca\") pod \"1ff2474e-cdfe-45f2-9326-28084a6aed91\" (UID: \"1ff2474e-cdfe-45f2-9326-28084a6aed91\") "
Nov 02 12:41:17.866709 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.835490    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:41:17.866709 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.819953    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.883022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.880618    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.886699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.885941    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:17.887091 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.887061    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-openshift-ca" pod "1ff2474e-cdfe-45f2-9326-28084a6aed91" (UID: "1ff2474e-cdfe-45f2-9326-28084a6aed91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:41:17.978020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:17.966204    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.106024 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.033179    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:41:18.191110 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.183153    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-xks-ca" pod "1ff2474e-cdfe-45f2-9326-28084a6aed91" (UID: "1ff2474e-cdfe-45f2-9326-28084a6aed91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:41:18.191110 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.183169    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff2474e-cdfe-45f2-9326-28084a6aed91-kube-api-access-gsphf" pod "1ff2474e-cdfe-45f2-9326-28084a6aed91" (UID: "1ff2474e-cdfe-45f2-9326-28084a6aed91"). InnerVolumeSpecName "kube-api-access-gsphf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:41:18.252148 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.224539    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsphf\" (UniqueName: \"kubernetes.io/projected/1ff2474e-cdfe-45f2-9326-28084a6aed91-kube-api-access-gsphf\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:41:18.252568 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.252534    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff2474e-cdfe-45f2-9326-28084a6aed91-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:41:18.252725 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.244383    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.438022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.408258    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 02 12:41:18.438022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.409300    5879 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Nov 02 12:41:18.489400 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.489290    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 12:41:18.489712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.489661    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 12:41:18.673080 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.673048    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"
Nov 02 12:41:18.673523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.673494    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6" event={"ID":"1ff2474e-cdfe-45f2-9326-28084a6aed91","Type":"ContainerDied","Data":"4775d4a6952bcd97abf46cee8b03ce8e85166eba7015df7e5305f7191b158372"}
Nov 02 12:41:18.673725 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.673707    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4775d4a6952bcd97abf46cee8b03ce8e85166eba7015df7e5305f7191b158372"
Nov 02 12:41:18.752370 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.752240    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:41:18.753548 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.752978    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 12:41:18.821480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.821124    5879 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.847551 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.847500    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 12:41:18.885890 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.884097    5879 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.890899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.887048    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.890899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.888145    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:18.890899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.888813    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 12:41:18.890899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:18.890284    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.122325 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.122277    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.264156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.264094    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.411211 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.411112    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.485283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.485233    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 12:41:19.485627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.485594    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 12:41:19.710622 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.710531    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:41:19.711166 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.711136    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 12:41:19.849517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.849459    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.888150 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.888105    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.889141 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.889121    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:41:19.903450 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.903412    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:19.903976 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.903947    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 12:41:19.910591 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:19.910554    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 12:41:20.059122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.059025    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:20.101902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.101856    5879 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.123612 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.123147    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.277242 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.277194    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.424017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.423065    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.433026 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.431502    5879 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.493019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.491517    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 12:41:20.493019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.491574    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 12:41:20.594148 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.594092    5879 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded"
Nov 02 12:41:20.751079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.745933    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:41:20.751079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.746007    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 12:41:20.862020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.852079    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.868019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.864950    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:20.908136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.908092    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.908733 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.908706    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 12:41:20.914128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.914076    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:20.914677 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:20.914649    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 12:41:21.126147 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.126099    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:21.283026 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.278117    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:41:21.380263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.373692    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:41:21.424219 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.424175    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 02 12:41:21.529030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.528304    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 12:41:21.545266 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.530825    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 12:41:21.545266 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.530877    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 12:41:21.570230 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.569373    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:41:21.678092 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.677025    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:41:21.686655 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.685368    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:41:21.687869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.687105    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:41:21.712775 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.712352    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:41:21.712775 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.712413    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 12:41:21.755114 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.755086    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:41:21.767922 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:21.767894    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:41:22.362182 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:22.362138    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:22.486879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:22.486428    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 12:41:22.486879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:22.486491    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 12:41:22.714019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:22.710409    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:41:22.714019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:22.710471    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 12:41:23.486457 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:23.486419    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 12:41:23.487096 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:23.487062    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 12:41:23.735017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:23.713427    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:41:23.735332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:23.735295    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 12:41:24.050314 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.050285    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"]
Nov 02 12:41:24.050812 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.050799    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1ff2474e-cdfe-45f2-9326-28084a6aed91" containerName="revoke-keys"
Nov 02 12:41:24.050868 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.050813    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff2474e-cdfe-45f2-9326-28084a6aed91" containerName="revoke-keys"
Nov 02 12:41:24.050934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.050925    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff2474e-cdfe-45f2-9326-28084a6aed91" containerName="revoke-keys"
Nov 02 12:41:24.056097 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.056049    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"]
Nov 02 12:41:24.056340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.056176    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:24.072880 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.072853    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:41:24.073071 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.072972    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:41:24.136271 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.136235    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aae62d2-r4mv4\" (UID: \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:24.136570 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.136550    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzn26\" (UniqueName: \"kubernetes.io/projected/232958a9-5f0e-4c72-9493-8f04d1ae6b50-kube-api-access-tzn26\") pod \"maas-api-revoke-sub-2aae62d2-r4mv4\" (UID: \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:24.136966 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.136943    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aae62d2-r4mv4\" (UID: \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:24.175066 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.175036    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:24.238113 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.238074    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzn26\" (UniqueName: \"kubernetes.io/projected/232958a9-5f0e-4c72-9493-8f04d1ae6b50-kube-api-access-tzn26\") pod \"maas-api-revoke-sub-2aae62d2-r4mv4\" (UID: \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:24.238360 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.238342    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aae62d2-r4mv4\" (UID: \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:24.238436 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.238422    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aae62d2-r4mv4\" (UID: \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:24.239303 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.239276    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2aae62d2-r4mv4\" (UID: \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:24.241011 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.240977    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2aae62d2-r4mv4\" (UID: \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:24.263254 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.263218    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzn26\" (UniqueName: \"kubernetes.io/projected/232958a9-5f0e-4c72-9493-8f04d1ae6b50-kube-api-access-tzn26\") pod \"maas-api-revoke-sub-2aae62d2-r4mv4\" (UID: \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:24.371535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.371493    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:24.498047 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.496469    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 12:41:24.498047 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.496534    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 12:41:24.711116 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.710983    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 12:41:24.711418 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.711386    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 12:41:24.725434 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:41:24.725394    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232958a9_5f0e_4c72_9493_8f04d1ae6b50.slice/crio-080b7f37dbeb58236fe1f69ecaa1b3457e43bc70057ed72b97c3d60cf0aed00e WatchSource:0}: Error finding container 080b7f37dbeb58236fe1f69ecaa1b3457e43bc70057ed72b97c3d60cf0aed00e: Status 404 returned error can't find the container with id 080b7f37dbeb58236fe1f69ecaa1b3457e43bc70057ed72b97c3d60cf0aed00e
Nov 02 12:41:24.727464 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.727372    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"]
Nov 02 12:41:24.841929 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:24.841893    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4" event={"ID":"232958a9-5f0e-4c72-9493-8f04d1ae6b50","Type":"ContainerStarted","Data":"080b7f37dbeb58236fe1f69ecaa1b3457e43bc70057ed72b97c3d60cf0aed00e"}
Nov 02 12:41:25.021571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:25.021469    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:25.486827 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:25.486786    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body=
Nov 02 12:41:25.487067 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:25.486851    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused"
Nov 02 12:41:25.572605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:25.572558    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:25.806444 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:25.806121    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:25.861278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:25.860641    5879 generic.go:356] "Generic (PLEG): container finished" podID="232958a9-5f0e-4c72-9493-8f04d1ae6b50" containerID="601f32d67358a0d1cd26036e2db730d41ddf6dec348ddab86ffad24766b4ee37" exitCode=0
Nov 02 12:41:25.861278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:25.860808    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4" event={"ID":"232958a9-5f0e-4c72-9493-8f04d1ae6b50","Type":"ContainerDied","Data":"601f32d67358a0d1cd26036e2db730d41ddf6dec348ddab86ffad24766b4ee37"}
Nov 02 12:41:25.959455 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:25.959419    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:25.960179 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:25.960124    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:25.961745 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:25.961685    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:41:26.238375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:26.238336    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:26.491386 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:26.491298    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:26.491821 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:26.491784    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:26.492793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:26.492774    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 12:41:27.039883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.039854    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:27.218699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.218570    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232958a9-5f0e-4c72-9493-8f04d1ae6b50-kube-api-access-tzn26\" (UniqueName: \"kubernetes.io/projected/232958a9-5f0e-4c72-9493-8f04d1ae6b50-kube-api-access-tzn26\") pod \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\" (UID: \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\") "
Nov 02 12:41:27.218699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.218681    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-xks-ca\") pod \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\" (UID: \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\") "
Nov 02 12:41:27.219283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.218777    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-openshift-ca\") pod \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\" (UID: \"232958a9-5f0e-4c72-9493-8f04d1ae6b50\") "
Nov 02 12:41:27.219750 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.219722    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-openshift-ca" pod "232958a9-5f0e-4c72-9493-8f04d1ae6b50" (UID: "232958a9-5f0e-4c72-9493-8f04d1ae6b50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:41:27.223323 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.223293    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-xks-ca" pod "232958a9-5f0e-4c72-9493-8f04d1ae6b50" (UID: "232958a9-5f0e-4c72-9493-8f04d1ae6b50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:41:27.223797 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.223773    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232958a9-5f0e-4c72-9493-8f04d1ae6b50-kube-api-access-tzn26" pod "232958a9-5f0e-4c72-9493-8f04d1ae6b50" (UID: "232958a9-5f0e-4c72-9493-8f04d1ae6b50"). InnerVolumeSpecName "kube-api-access-tzn26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:41:27.320285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.320236    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:41:27.320285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.320272    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzn26\" (UniqueName: \"kubernetes.io/projected/232958a9-5f0e-4c72-9493-8f04d1ae6b50-kube-api-access-tzn26\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:41:27.320285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.320286    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232958a9-5f0e-4c72-9493-8f04d1ae6b50-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:41:27.875653 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.875622    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"
Nov 02 12:41:27.877884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.875653    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4" event={"ID":"232958a9-5f0e-4c72-9493-8f04d1ae6b50","Type":"ContainerDied","Data":"080b7f37dbeb58236fe1f69ecaa1b3457e43bc70057ed72b97c3d60cf0aed00e"}
Nov 02 12:41:27.877884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.875709    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="080b7f37dbeb58236fe1f69ecaa1b3457e43bc70057ed72b97c3d60cf0aed00e"
Nov 02 12:41:27.944183 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:27.944155    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:29.404690 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:29.404661    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:29.525663 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:29.525633    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:29.705333 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:29.705221    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:31.256122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:31.256083    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:31.377497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:31.377458    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:31.576547 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:31.576463    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:33.859381 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:33.858538    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:34.461578 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:34.461536    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:34.934018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:34.923407    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"]
Nov 02 12:41:34.934018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:34.924460    5879 cpu_manager.go:414] "containerMap: removing container" podUID="232958a9-5f0e-4c72-9493-8f04d1ae6b50" containerName="revoke-keys"
Nov 02 12:41:34.934018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:34.924481    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232958a9-5f0e-4c72-9493-8f04d1ae6b50" containerName="revoke-keys"
Nov 02 12:41:34.934018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:34.924639    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="232958a9-5f0e-4c72-9493-8f04d1ae6b50" containerName="revoke-keys"
Nov 02 12:41:34.943573 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:34.943545    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:34.947762 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:34.947730    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"]
Nov 02 12:41:34.954335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:34.954288    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"]
Nov 02 12:41:34.954724 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:34.954704    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.044800    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.044856    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a14733e-c82f-4cae-8378-f87260f4b338-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.044912    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9x4h\" (UniqueName: \"kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-kube-api-access-w9x4h\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.044938    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbjwk\" (UniqueName: \"kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-kube-api-access-mbjwk\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.044971    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045046    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045085    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045175    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a14733e-c82f-4cae-8378-f87260f4b338-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045238    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045278    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045322    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/143c9cce-2a0f-454d-b363-87ebf116e254-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045364    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045432    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045484    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045518    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/143c9cce-2a0f-454d-b363-87ebf116e254-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045553    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045633    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.050262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.045664    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.082109 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.082025    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"]
Nov 02 12:41:35.147964 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.147918    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.149167 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149124    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/143c9cce-2a0f-454d-b363-87ebf116e254-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.149385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149188    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.149385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149272    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.149385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149305    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.149385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149358    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.149385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149384    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a14733e-c82f-4cae-8378-f87260f4b338-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.149696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149440    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9x4h\" (UniqueName: \"kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-kube-api-access-w9x4h\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.149696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149467    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbjwk\" (UniqueName: \"kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-kube-api-access-mbjwk\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.149696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149498    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.149696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149576    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.149696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149623    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.149973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149725    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a14733e-c82f-4cae-8378-f87260f4b338-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.149973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149798    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.149973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149835    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.149973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149884    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/143c9cce-2a0f-454d-b363-87ebf116e254-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.149973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.149931    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.150280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.150024    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.150667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.150635    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.151342 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.151308    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.154032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.152824    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.154032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.153179    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.154032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.153843    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a14733e-c82f-4cae-8378-f87260f4b338-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.155605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.154684    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.155605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.155045    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.155605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.155298    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.155605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.155341    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/143c9cce-2a0f-454d-b363-87ebf116e254-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.156480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.156169    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.156696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.156594    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/143c9cce-2a0f-454d-b363-87ebf116e254-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.161283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.161213    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a14733e-c82f-4cae-8378-f87260f4b338-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.163893 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.163831    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.164309 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.164253    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.183419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.182515    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.190832 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.190707    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.191747 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.191687    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbjwk\" (UniqueName: \"kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-kube-api-access-mbjwk\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.195155 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.195102    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9x4h\" (UniqueName: \"kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-kube-api-access-w9x4h\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.379968 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.379934    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:35.382949 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.382919    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:35.710413 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.710288    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:35.719715 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:41:35.719684    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143c9cce_2a0f_454d_b363_87ebf116e254.slice/crio-c1cbd65f243e3ff54565f4b4fd2a79174c936a841edaab034e1b43063cfc667a WatchSource:0}: Error finding container c1cbd65f243e3ff54565f4b4fd2a79174c936a841edaab034e1b43063cfc667a: Status 404 returned error can't find the container with id c1cbd65f243e3ff54565f4b4fd2a79174c936a841edaab034e1b43063cfc667a
Nov 02 12:41:35.721351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.721324    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"]
Nov 02 12:41:35.722450 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.722417    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:35.722514 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.722484    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:35.722554 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.722513    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:35.732982 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:41:35.732948    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a14733e_c82f_4cae_8378_f87260f4b338.slice/crio-d5a28f9f21775a1b6a20777ebe8bcb203c47accbd68060f4298d016000dbfd44 WatchSource:0}: Error finding container d5a28f9f21775a1b6a20777ebe8bcb203c47accbd68060f4298d016000dbfd44: Status 404 returned error can't find the container with id d5a28f9f21775a1b6a20777ebe8bcb203c47accbd68060f4298d016000dbfd44
Nov 02 12:41:35.733854 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.733833    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"]
Nov 02 12:41:35.735494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.735458    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:35.735600 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.735540    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:35.735664 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.735594    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:35.840554 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:35.840476    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:36.112535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.112445    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" event={"ID":"143c9cce-2a0f-454d-b363-87ebf116e254","Type":"ContainerStarted","Data":"da43298ba95d9f2323bfc806749b4e0d2e87fce74736868f9a51db4fea2b813c"}
Nov 02 12:41:36.112535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.112510    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" event={"ID":"143c9cce-2a0f-454d-b363-87ebf116e254","Type":"ContainerStarted","Data":"c1cbd65f243e3ff54565f4b4fd2a79174c936a841edaab034e1b43063cfc667a"}
Nov 02 12:41:36.113977 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.113937    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" event={"ID":"0a14733e-c82f-4cae-8378-f87260f4b338","Type":"ContainerStarted","Data":"7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859"}
Nov 02 12:41:36.114172 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.114012    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" event={"ID":"0a14733e-c82f-4cae-8378-f87260f4b338","Type":"ContainerStarted","Data":"d5a28f9f21775a1b6a20777ebe8bcb203c47accbd68060f4298d016000dbfd44"}
Nov 02 12:41:36.178018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.177063    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" podStartSLOduration=2.177044451 podStartE2EDuration="2.177044451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:36.171359152 +0000 UTC m=+842.278062005" watchObservedRunningTime="2026-11-02 12:41:36.177044451 +0000 UTC m=+842.283747302"
Nov 02 12:41:36.198552 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.198503    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:36.227016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.226730    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" podStartSLOduration=2.226707319 podStartE2EDuration="2.226707319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:36.218811149 +0000 UTC m=+842.325514001" watchObservedRunningTime="2026-11-02 12:41:36.226707319 +0000 UTC m=+842.333410172"
Nov 02 12:41:36.351619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.351586    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:36.380693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.380656    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:36.382531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.382497    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:41:36.382707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.382567    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" podUID="143c9cce-2a0f-454d-b363-87ebf116e254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 12:41:36.383224 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.383201    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:36.385042 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.384890    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:41:36.385042 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:36.384948    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" podUID="0a14733e-c82f-4cae-8378-f87260f4b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 12:41:37.380287 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:37.380256    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:41:37.380690 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:37.380315    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" podUID="143c9cce-2a0f-454d-b363-87ebf116e254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 12:41:37.391017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:37.385459    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:41:37.391017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:37.385538    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" podUID="0a14733e-c82f-4cae-8378-f87260f4b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 12:41:37.561981 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:37.561950    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:38.384681 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:38.383345    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:41:38.384681 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:38.383407    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" podUID="0a14733e-c82f-4cae-8378-f87260f4b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 12:41:38.385601 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:38.385578    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:41:38.385774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:38.385754    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" podUID="143c9cce-2a0f-454d-b363-87ebf116e254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 12:41:39.380592 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:39.380556    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:41:39.380775 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:39.380609    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" podUID="143c9cce-2a0f-454d-b363-87ebf116e254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 12:41:39.393023 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:39.385355    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:41:39.393023 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:39.385411    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" podUID="0a14733e-c82f-4cae-8378-f87260f4b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 12:41:39.544285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:39.544244    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:40.021020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:40.020662    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:40.146115 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:40.146065    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:40.380296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:40.380263    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:41:40.380620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:40.380585    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" podUID="143c9cce-2a0f-454d-b363-87ebf116e254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 12:41:40.385390 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:40.385360    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:41:40.385595 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:40.385566    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" podUID="0a14733e-c82f-4cae-8378-f87260f4b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 12:41:41.380847 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:41.380801    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:41:41.381400 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:41.380889    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" podUID="143c9cce-2a0f-454d-b363-87ebf116e254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 12:41:41.387031 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:41.385371    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:41:41.387031 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:41.385427    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" podUID="0a14733e-c82f-4cae-8378-f87260f4b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 12:41:41.773948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:41.773860    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:41.937864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:41.937827    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:42.146013 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:42.145540    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:42.380653 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:42.380613    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 12:41:42.380839 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:42.380702    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" podUID="143c9cce-2a0f-454d-b363-87ebf116e254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 12:41:42.384013 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:42.383969    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 12:41:42.384376 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:42.384045    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" podUID="0a14733e-c82f-4cae-8378-f87260f4b338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 12:41:43.661222 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:43.661175    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:43.729919 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:43.729892    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:43.730566 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:43.730518    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:43.731646 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:43.731614    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:41:43.759959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:43.759921    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:43.760629 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:43.760605    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:43.761510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:43.761493    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:41:44.822135 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:41:44.822098    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:41:45.323912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:45.323874    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:46.036199 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.036163    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"]
Nov 02 12:41:46.041882 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.041856    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.082583 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.082552    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"]
Nov 02 12:41:46.163314 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.163280    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.163494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.163429    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.163494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.163458    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.163571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.163508    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.163637 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.163614    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11648a63-a758-4f5e-be54-976d4e234612-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.163684 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.163669    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knn7v\" (UniqueName: \"kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-kube-api-access-knn7v\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.163772 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.163756    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11648a63-a758-4f5e-be54-976d4e234612-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.163823 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.163810    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.163859 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.163848    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.265086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.265035    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11648a63-a758-4f5e-be54-976d4e234612-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.265311 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.265100    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.265311 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.265148    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.265311 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.265274    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.265498 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.265415    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.265498 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.265468    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.265601 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.265539    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.265601 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.265583    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11648a63-a758-4f5e-be54-976d4e234612-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.265707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.265637    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knn7v\" (UniqueName: \"kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-kube-api-access-knn7v\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.265961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.265930    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.266288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.265957    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.266288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.266137    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.266288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.266210    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.266491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.266333    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11648a63-a758-4f5e-be54-976d4e234612-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.267878 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.267851    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.267984 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.267954    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11648a63-a758-4f5e-be54-976d4e234612-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.290385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.290328    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.290561 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.290544    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knn7v\" (UniqueName: \"kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-kube-api-access-knn7v\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.359777 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.359743    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:46.681132 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:41:46.681100    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11648a63_a758_4f5e_be54_976d4e234612.slice/crio-1da9355081f2307cf58f52f23d1ce293b1331dcaa1102bc3df70971baeadd501 WatchSource:0}: Error finding container 1da9355081f2307cf58f52f23d1ce293b1331dcaa1102bc3df70971baeadd501: Status 404 returned error can't find the container with id 1da9355081f2307cf58f52f23d1ce293b1331dcaa1102bc3df70971baeadd501
Nov 02 12:41:46.683011 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.682972    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"]
Nov 02 12:41:46.683661 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.683626    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:46.683709 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.683695    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:46.683745 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:46.683727    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:41:47.099924 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:47.099799    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:47.261614 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:47.261581    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" event={"ID":"11648a63-a758-4f5e-be54-976d4e234612","Type":"ContainerStarted","Data":"67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750"}
Nov 02 12:41:47.261830 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:47.261641    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" event={"ID":"11648a63-a758-4f5e-be54-976d4e234612","Type":"ContainerStarted","Data":"1da9355081f2307cf58f52f23d1ce293b1331dcaa1102bc3df70971baeadd501"}
Nov 02 12:41:47.360591 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:47.360495    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:47.362104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:47.362083    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 12:41:47.362211 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:47.362130    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" podUID="11648a63-a758-4f5e-be54-976d4e234612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 12:41:48.360320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:48.360288    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 12:41:48.360716 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:48.360339    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" podUID="11648a63-a758-4f5e-be54-976d4e234612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 12:41:48.770856 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:48.770822    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:49.360161 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:49.360126    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 12:41:49.360370 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:49.360186    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" podUID="11648a63-a758-4f5e-be54-976d4e234612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 12:41:50.361025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:50.360957    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 12:41:50.361437 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:50.361055    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" podUID="11648a63-a758-4f5e-be54-976d4e234612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 12:41:50.475822 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:50.475792    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:51.360835 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:51.360801    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 12:41:51.361087 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:51.360861    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" podUID="11648a63-a758-4f5e-be54-976d4e234612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 12:41:52.137879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:52.137847    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:52.262311 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:52.262282    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:52.387018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:52.386354    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 12:41:52.387018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:52.386435    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" podUID="11648a63-a758-4f5e-be54-976d4e234612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 12:41:52.409077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:52.408313    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:53.360335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:53.360295    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 02 12:41:53.360640 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:53.360606    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" podUID="11648a63-a758-4f5e-be54-976d4e234612" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 02 12:41:54.070814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:54.070778    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:54.744556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:54.744526    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:54.753693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:54.753661    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:54.756530 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:54.756511    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:41:54.840262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:54.840171    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" podStartSLOduration=9.840147584 podStartE2EDuration="9.840147584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:41:47.336922353 +0000 UTC m=+853.443625204" watchObservedRunningTime="2026-11-02 12:41:54.840147584 +0000 UTC m=+860.946850437"
Nov 02 12:41:55.847186 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:55.847115    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:57.456693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:57.456657    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:58.999627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:58.999594    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:41:59.674242 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:41:59.674208    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:00.611039 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:00.610978    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:01.995826 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:01.995786    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"]
Nov 02 12:42:02.000516 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.000495    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.020735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.020701    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 12:42:02.020735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.020701    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 12:42:02.020933 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.020702    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 12:42:02.020933 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.020861    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6fgxx\""
Nov 02 12:42:02.041022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.040952    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"]
Nov 02 12:42:02.084948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.084920    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.085132 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.084953    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.085188 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.085163    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.085288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.085273    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpztn\" (UniqueName: \"kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-kube-api-access-qpztn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.085328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.085312    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.085364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.085334    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/280bc232-ca40-4246-ac92-5cda8dac9930-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.085418 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.085405    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.186720 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.186683    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpztn\" (UniqueName: \"kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-kube-api-access-qpztn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.186895 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.186758    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.186895 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.186796    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/280bc232-ca40-4246-ac92-5cda8dac9930-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.186895 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.186831    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.187067 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.186917    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.187067 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.186946    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.187181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.187111    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.187549 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.187522    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.190068 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.189985    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/280bc232-ca40-4246-ac92-5cda8dac9930-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.190068 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.190048    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.190542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.190522    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.190691 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.190656    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.203177 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.203141    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:02.212167 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.212137    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpztn\" (UniqueName: \"kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-kube-api-access-qpztn\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.212320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.212189    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vzlw6\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.315492 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.315407    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:02.407256 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.407156    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:02.659977 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:42:02.659933    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280bc232_ca40_4246_ac92_5cda8dac9930.slice/crio-4bc10f01314793438cb1fcf83e8bd4939ba5c0f15da5e5653e5268a2e2c5eb35 WatchSource:0}: Error finding container 4bc10f01314793438cb1fcf83e8bd4939ba5c0f15da5e5653e5268a2e2c5eb35: Status 404 returned error can't find the container with id 4bc10f01314793438cb1fcf83e8bd4939ba5c0f15da5e5653e5268a2e2c5eb35
Nov 02 12:42:02.661365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.661342    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"]
Nov 02 12:42:02.662276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:02.662257    5879 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 12:42:03.444276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:03.444242    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6" event={"ID":"280bc232-ca40-4246-ac92-5cda8dac9930","Type":"ContainerStarted","Data":"4bc10f01314793438cb1fcf83e8bd4939ba5c0f15da5e5653e5268a2e2c5eb35"}
Nov 02 12:42:03.978014 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:03.977959    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:04.928455 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:04.928401    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"]
Nov 02 12:42:04.938901 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:04.938876    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:04.953603 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:04.953569    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"]
Nov 02 12:42:04.960909 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:04.960845    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:42:04.961313 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:04.961293    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:42:05.067228 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.067197    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0187974-f552-49ad-a0d2-752794a67267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-496b16f1-9d56h\" (UID: \"c0187974-f552-49ad-a0d2-752794a67267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:05.067523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.067501    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0187974-f552-49ad-a0d2-752794a67267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-496b16f1-9d56h\" (UID: \"c0187974-f552-49ad-a0d2-752794a67267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:05.067620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.067561    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7twq\" (UniqueName: \"kubernetes.io/projected/c0187974-f552-49ad-a0d2-752794a67267-kube-api-access-s7twq\") pod \"maas-api-revoke-sub-496b16f1-9d56h\" (UID: \"c0187974-f552-49ad-a0d2-752794a67267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:05.169261 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.169208    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0187974-f552-49ad-a0d2-752794a67267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-496b16f1-9d56h\" (UID: \"c0187974-f552-49ad-a0d2-752794a67267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:05.169462 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.169293    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7twq\" (UniqueName: \"kubernetes.io/projected/c0187974-f552-49ad-a0d2-752794a67267-kube-api-access-s7twq\") pod \"maas-api-revoke-sub-496b16f1-9d56h\" (UID: \"c0187974-f552-49ad-a0d2-752794a67267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:05.169462 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.169357    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0187974-f552-49ad-a0d2-752794a67267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-496b16f1-9d56h\" (UID: \"c0187974-f552-49ad-a0d2-752794a67267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:05.170984 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.170920    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0187974-f552-49ad-a0d2-752794a67267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-496b16f1-9d56h\" (UID: \"c0187974-f552-49ad-a0d2-752794a67267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:05.172687 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.172661    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0187974-f552-49ad-a0d2-752794a67267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-496b16f1-9d56h\" (UID: \"c0187974-f552-49ad-a0d2-752794a67267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:05.195386 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.195312    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7twq\" (UniqueName: \"kubernetes.io/projected/c0187974-f552-49ad-a0d2-752794a67267-kube-api-access-s7twq\") pod \"maas-api-revoke-sub-496b16f1-9d56h\" (UID: \"c0187974-f552-49ad-a0d2-752794a67267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:05.259003 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.258967    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:05.584142 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:42:05.584112    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0187974_f552_49ad_a0d2_752794a67267.slice/crio-6343e5889388053ff46f319400274cd2c066e7c6857a06a80671876d1fcf8709 WatchSource:0}: Error finding container 6343e5889388053ff46f319400274cd2c066e7c6857a06a80671876d1fcf8709: Status 404 returned error can't find the container with id 6343e5889388053ff46f319400274cd2c066e7c6857a06a80671876d1fcf8709
Nov 02 12:42:05.585868 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.585822    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"]
Nov 02 12:42:05.593358 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.592226    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:05.688263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.688228    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:42:05.688395 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:05.688302    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:42:06.489552 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:06.489502    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6" event={"ID":"280bc232-ca40-4246-ac92-5cda8dac9930","Type":"ContainerStarted","Data":"6767b8fe8de377acd36d989bade1de492d7a7dc3b5e6aae6fda9578d266219e3"}
Nov 02 12:42:06.490276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:06.490251    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:06.492587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:06.492491    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h" event={"ID":"c0187974-f552-49ad-a0d2-752794a67267","Type":"ContainerStarted","Data":"baec456dd7457abe8f6f10cf773a48b9875f0cf205f56e69813df082834342e7"}
Nov 02 12:42:06.492587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:06.492550    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h" event={"ID":"c0187974-f552-49ad-a0d2-752794a67267","Type":"ContainerStarted","Data":"6343e5889388053ff46f319400274cd2c066e7c6857a06a80671876d1fcf8709"}
Nov 02 12:42:06.492587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:06.492573    5879 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-vzlw6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:42:06.492826 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:06.492623    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6" podUID="280bc232-ca40-4246-ac92-5cda8dac9930" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:42:06.554060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:06.551939    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6" podStartSLOduration=2.52729757 podStartE2EDuration="5.551918853s" totalImagesPullingTime="3.024621283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:06.550542232 +0000 UTC m=+872.657245089" watchObservedRunningTime="2026-11-02 12:42:06.551918853 +0000 UTC m=+872.658621712"
Nov 02 12:42:06.599768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:06.599710    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h" podStartSLOduration=2.599696327 podStartE2EDuration="2.599696327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:06.595847885 +0000 UTC m=+872.702550736" watchObservedRunningTime="2026-11-02 12:42:06.599696327 +0000 UTC m=+872.706399178"
Nov 02 12:42:07.168289 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:07.168250    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:07.501083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:07.500962    5879 generic.go:356] "Generic (PLEG): container finished" podID="c0187974-f552-49ad-a0d2-752794a67267" containerID="baec456dd7457abe8f6f10cf773a48b9875f0cf205f56e69813df082834342e7" exitCode=0
Nov 02 12:42:07.501083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:07.501056    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h" event={"ID":"c0187974-f552-49ad-a0d2-752794a67267","Type":"ContainerDied","Data":"baec456dd7457abe8f6f10cf773a48b9875f0cf205f56e69813df082834342e7"}
Nov 02 12:42:07.502068 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:07.502041    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:42:08.681781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:08.681754    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:08.768042 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:08.767841    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:08.828442 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:08.828391    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0187974-f552-49ad-a0d2-752794a67267-kube-api-access-s7twq\" (UniqueName: \"kubernetes.io/projected/c0187974-f552-49ad-a0d2-752794a67267-kube-api-access-s7twq\") pod \"c0187974-f552-49ad-a0d2-752794a67267\" (UID: \"c0187974-f552-49ad-a0d2-752794a67267\") "
Nov 02 12:42:08.828693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:08.828677    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0187974-f552-49ad-a0d2-752794a67267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0187974-f552-49ad-a0d2-752794a67267-maas-api-openshift-ca\") pod \"c0187974-f552-49ad-a0d2-752794a67267\" (UID: \"c0187974-f552-49ad-a0d2-752794a67267\") "
Nov 02 12:42:08.828879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:08.828755    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0187974-f552-49ad-a0d2-752794a67267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0187974-f552-49ad-a0d2-752794a67267-maas-api-xks-ca\") pod \"c0187974-f552-49ad-a0d2-752794a67267\" (UID: \"c0187974-f552-49ad-a0d2-752794a67267\") "
Nov 02 12:42:08.829044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:08.829019    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0187974-f552-49ad-a0d2-752794a67267-maas-api-openshift-ca" pod "c0187974-f552-49ad-a0d2-752794a67267" (UID: "c0187974-f552-49ad-a0d2-752794a67267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:42:08.829416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:08.829389    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0187974-f552-49ad-a0d2-752794a67267-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:08.835212 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:08.834282    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0187974-f552-49ad-a0d2-752794a67267-kube-api-access-s7twq" pod "c0187974-f552-49ad-a0d2-752794a67267" (UID: "c0187974-f552-49ad-a0d2-752794a67267"). InnerVolumeSpecName "kube-api-access-s7twq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:42:08.835212 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:08.835617    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0187974-f552-49ad-a0d2-752794a67267-maas-api-xks-ca" pod "c0187974-f552-49ad-a0d2-752794a67267" (UID: "c0187974-f552-49ad-a0d2-752794a67267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:42:08.931239 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:08.931201    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0187974-f552-49ad-a0d2-752794a67267-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:08.931413 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:08.931230    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7twq\" (UniqueName: \"kubernetes.io/projected/c0187974-f552-49ad-a0d2-752794a67267-kube-api-access-s7twq\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:09.520970 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:09.520941    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"
Nov 02 12:42:09.520970 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:09.520958    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h" event={"ID":"c0187974-f552-49ad-a0d2-752794a67267","Type":"ContainerDied","Data":"6343e5889388053ff46f319400274cd2c066e7c6857a06a80671876d1fcf8709"}
Nov 02 12:42:09.521261 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:09.521025    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6343e5889388053ff46f319400274cd2c066e7c6857a06a80671876d1fcf8709"
Nov 02 12:42:10.274751 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:10.274713    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:10.459219 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:10.459189    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:10.927330 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:10.927290    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:11.376339 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:11.376305    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:11.709548 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:11.709438    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:12.160101 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:12.160065    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:13.794435 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:13.794381    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:14.701869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:14.701837    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:14.822433 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:42:14.822395    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:42:14.833820 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:14.833796    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:15.713043 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:15.712977    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:16.909191 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:16.909155    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:17.048276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:17.048234    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:17.615675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:17.615630    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:19.663754 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:19.663651    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:20.100675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.100578    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"]
Nov 02 12:42:20.101245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.101228    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c0187974-f552-49ad-a0d2-752794a67267" containerName="revoke-keys"
Nov 02 12:42:20.101337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.101248    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0187974-f552-49ad-a0d2-752794a67267" containerName="revoke-keys"
Nov 02 12:42:20.101421 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.101410    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0187974-f552-49ad-a0d2-752794a67267" containerName="revoke-keys"
Nov 02 12:42:20.104885 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.104868    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:20.125914 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.125886    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"]
Nov 02 12:42:20.134802 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.134776    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:42:20.134802 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.134794    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:42:20.186344 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.186294    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdlr2\" (UniqueName: \"kubernetes.io/projected/93f7bc99-e37d-40c0-8948-aca0afb569de-kube-api-access-mdlr2\") pod \"maas-api-revoke-sub-930d4880-drqxd\" (UID: \"93f7bc99-e37d-40c0-8948-aca0afb569de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:20.186520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.186443    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-930d4880-drqxd\" (UID: \"93f7bc99-e37d-40c0-8948-aca0afb569de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:20.186520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.186480    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-930d4880-drqxd\" (UID: \"93f7bc99-e37d-40c0-8948-aca0afb569de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:20.287409 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.287369    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdlr2\" (UniqueName: \"kubernetes.io/projected/93f7bc99-e37d-40c0-8948-aca0afb569de-kube-api-access-mdlr2\") pod \"maas-api-revoke-sub-930d4880-drqxd\" (UID: \"93f7bc99-e37d-40c0-8948-aca0afb569de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:20.287620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.287502    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-930d4880-drqxd\" (UID: \"93f7bc99-e37d-40c0-8948-aca0afb569de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:20.287620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.287559    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-930d4880-drqxd\" (UID: \"93f7bc99-e37d-40c0-8948-aca0afb569de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:20.288314 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.288287    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-930d4880-drqxd\" (UID: \"93f7bc99-e37d-40c0-8948-aca0afb569de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:20.290277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.290255    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-930d4880-drqxd\" (UID: \"93f7bc99-e37d-40c0-8948-aca0afb569de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:20.312873 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.312844    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdlr2\" (UniqueName: \"kubernetes.io/projected/93f7bc99-e37d-40c0-8948-aca0afb569de-kube-api-access-mdlr2\") pod \"maas-api-revoke-sub-930d4880-drqxd\" (UID: \"93f7bc99-e37d-40c0-8948-aca0afb569de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:20.421746 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.421702    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:20.734529 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:42:20.734495    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f7bc99_e37d_40c0_8948_aca0afb569de.slice/crio-e806b43836f6e17b90fa7c294e0cb34c0b40c7a36ad0d8f243fccdebddea3a60 WatchSource:0}: Error finding container e806b43836f6e17b90fa7c294e0cb34c0b40c7a36ad0d8f243fccdebddea3a60: Status 404 returned error can't find the container with id e806b43836f6e17b90fa7c294e0cb34c0b40c7a36ad0d8f243fccdebddea3a60
Nov 02 12:42:20.735526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:20.735504    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"]
Nov 02 12:42:21.369943 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:21.369910    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:21.695804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:21.695716    5879 generic.go:356] "Generic (PLEG): container finished" podID="93f7bc99-e37d-40c0-8948-aca0afb569de" containerID="887b2e31e3787634ae2951298c5b9c3353f56ae386190018e5f006188ea9c85d" exitCode=0
Nov 02 12:42:21.695952 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:21.695807    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd" event={"ID":"93f7bc99-e37d-40c0-8948-aca0afb569de","Type":"ContainerDied","Data":"887b2e31e3787634ae2951298c5b9c3353f56ae386190018e5f006188ea9c85d"}
Nov 02 12:42:21.695952 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:21.695861    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd" event={"ID":"93f7bc99-e37d-40c0-8948-aca0afb569de","Type":"ContainerStarted","Data":"e806b43836f6e17b90fa7c294e0cb34c0b40c7a36ad0d8f243fccdebddea3a60"}
Nov 02 12:42:22.871836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:22.871807    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:22.923229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:22.923198    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-xks-ca\") pod \"93f7bc99-e37d-40c0-8948-aca0afb569de\" (UID: \"93f7bc99-e37d-40c0-8948-aca0afb569de\") "
Nov 02 12:42:22.923396 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:22.923324    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-openshift-ca\") pod \"93f7bc99-e37d-40c0-8948-aca0afb569de\" (UID: \"93f7bc99-e37d-40c0-8948-aca0afb569de\") "
Nov 02 12:42:22.923607 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:22.923568    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f7bc99-e37d-40c0-8948-aca0afb569de-kube-api-access-mdlr2\" (UniqueName: \"kubernetes.io/projected/93f7bc99-e37d-40c0-8948-aca0afb569de-kube-api-access-mdlr2\") pod \"93f7bc99-e37d-40c0-8948-aca0afb569de\" (UID: \"93f7bc99-e37d-40c0-8948-aca0afb569de\") "
Nov 02 12:42:22.923784 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:22.923756    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-openshift-ca" pod "93f7bc99-e37d-40c0-8948-aca0afb569de" (UID: "93f7bc99-e37d-40c0-8948-aca0afb569de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:42:22.924172 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:22.924153    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:22.926229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:22.926202    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f7bc99-e37d-40c0-8948-aca0afb569de-kube-api-access-mdlr2" pod "93f7bc99-e37d-40c0-8948-aca0afb569de" (UID: "93f7bc99-e37d-40c0-8948-aca0afb569de"). InnerVolumeSpecName "kube-api-access-mdlr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:42:22.926644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:22.926620    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-xks-ca" pod "93f7bc99-e37d-40c0-8948-aca0afb569de" (UID: "93f7bc99-e37d-40c0-8948-aca0afb569de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:42:22.980336 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:22.980244    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:23.025896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:23.025859    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdlr2\" (UniqueName: \"kubernetes.io/projected/93f7bc99-e37d-40c0-8948-aca0afb569de-kube-api-access-mdlr2\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:23.025896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:23.025888    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f7bc99-e37d-40c0-8948-aca0afb569de-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:23.713243 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:23.713212    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"
Nov 02 12:42:23.713430 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:23.713258    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd" event={"ID":"93f7bc99-e37d-40c0-8948-aca0afb569de","Type":"ContainerDied","Data":"e806b43836f6e17b90fa7c294e0cb34c0b40c7a36ad0d8f243fccdebddea3a60"}
Nov 02 12:42:23.713430 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:23.713306    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e806b43836f6e17b90fa7c294e0cb34c0b40c7a36ad0d8f243fccdebddea3a60"
Nov 02 12:42:24.637127 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:24.637083    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:26.260699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:26.260655    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:27.918842 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:27.918784    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:28.044188 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:28.044131    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:28.176445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:28.176360    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:29.669175 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:29.669135    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:30.502414 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.502374    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"]
Nov 02 12:42:30.503058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.503039    5879 cpu_manager.go:414] "containerMap: removing container" podUID="93f7bc99-e37d-40c0-8948-aca0afb569de" containerName="revoke-keys"
Nov 02 12:42:30.503058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.503058    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f7bc99-e37d-40c0-8948-aca0afb569de" containerName="revoke-keys"
Nov 02 12:42:30.503272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.503259    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f7bc99-e37d-40c0-8948-aca0afb569de" containerName="revoke-keys"
Nov 02 12:42:30.507288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.507265    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:30.527549 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.527520    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"]
Nov 02 12:42:30.536308 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.536260    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:42:30.536467 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.536343    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:42:30.634062 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.634025    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a451697-26468\" (UID: \"84c5a86f-ba9c-43ee-8faa-4927990018da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:30.634263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.634186    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp769\" (UniqueName: \"kubernetes.io/projected/84c5a86f-ba9c-43ee-8faa-4927990018da-kube-api-access-fp769\") pod \"maas-api-revoke-sub-7a451697-26468\" (UID: \"84c5a86f-ba9c-43ee-8faa-4927990018da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:30.634324 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.634262    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a451697-26468\" (UID: \"84c5a86f-ba9c-43ee-8faa-4927990018da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:30.735469 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.735415    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a451697-26468\" (UID: \"84c5a86f-ba9c-43ee-8faa-4927990018da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:30.735874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.735527    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp769\" (UniqueName: \"kubernetes.io/projected/84c5a86f-ba9c-43ee-8faa-4927990018da-kube-api-access-fp769\") pod \"maas-api-revoke-sub-7a451697-26468\" (UID: \"84c5a86f-ba9c-43ee-8faa-4927990018da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:30.735874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.735559    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a451697-26468\" (UID: \"84c5a86f-ba9c-43ee-8faa-4927990018da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:30.736273 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.736255    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a451697-26468\" (UID: \"84c5a86f-ba9c-43ee-8faa-4927990018da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:30.738051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.738024    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a451697-26468\" (UID: \"84c5a86f-ba9c-43ee-8faa-4927990018da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:30.761463 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.761390    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp769\" (UniqueName: \"kubernetes.io/projected/84c5a86f-ba9c-43ee-8faa-4927990018da-kube-api-access-fp769\") pod \"maas-api-revoke-sub-7a451697-26468\" (UID: \"84c5a86f-ba9c-43ee-8faa-4927990018da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:30.822054 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:30.822023    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:31.130949 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:42:31.130913    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c5a86f_ba9c_43ee_8faa_4927990018da.slice/crio-a40a8eaab8b59d19271555c355bded148774c4493419d9a4ee14d50cd6bddf12 WatchSource:0}: Error finding container a40a8eaab8b59d19271555c355bded148774c4493419d9a4ee14d50cd6bddf12: Status 404 returned error can't find the container with id a40a8eaab8b59d19271555c355bded148774c4493419d9a4ee14d50cd6bddf12
Nov 02 12:42:31.131968 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:31.131939    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"]
Nov 02 12:42:31.348346 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:31.348306    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:31.788119 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:31.788029    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468" event={"ID":"84c5a86f-ba9c-43ee-8faa-4927990018da","Type":"ContainerStarted","Data":"ab447e75dc3447295c0c7ec42d338abd8d9c79cab762f0841bc3265d3b9d716c"}
Nov 02 12:42:31.788119 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:31.788076    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468" event={"ID":"84c5a86f-ba9c-43ee-8faa-4927990018da","Type":"ContainerStarted","Data":"a40a8eaab8b59d19271555c355bded148774c4493419d9a4ee14d50cd6bddf12"}
Nov 02 12:42:31.834043 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:31.833979    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468" podStartSLOduration=1.833964453 podStartE2EDuration="1.833964453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:31.831411591 +0000 UTC m=+897.938114447" watchObservedRunningTime="2026-11-02 12:42:31.833964453 +0000 UTC m=+897.940667304"
Nov 02 12:42:32.797118 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:32.797077    5879 generic.go:356] "Generic (PLEG): container finished" podID="84c5a86f-ba9c-43ee-8faa-4927990018da" containerID="ab447e75dc3447295c0c7ec42d338abd8d9c79cab762f0841bc3265d3b9d716c" exitCode=0
Nov 02 12:42:32.797546 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:32.797166    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468" event={"ID":"84c5a86f-ba9c-43ee-8faa-4927990018da","Type":"ContainerDied","Data":"ab447e75dc3447295c0c7ec42d338abd8d9c79cab762f0841bc3265d3b9d716c"}
Nov 02 12:42:33.051663 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:33.051574    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:34.041870 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.041841    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:34.188456 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.188425    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-xks-ca\") pod \"84c5a86f-ba9c-43ee-8faa-4927990018da\" (UID: \"84c5a86f-ba9c-43ee-8faa-4927990018da\") "
Nov 02 12:42:34.188654 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.188510    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-openshift-ca\") pod \"84c5a86f-ba9c-43ee-8faa-4927990018da\" (UID: \"84c5a86f-ba9c-43ee-8faa-4927990018da\") "
Nov 02 12:42:34.189132 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.188756    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c5a86f-ba9c-43ee-8faa-4927990018da-kube-api-access-fp769\" (UniqueName: \"kubernetes.io/projected/84c5a86f-ba9c-43ee-8faa-4927990018da-kube-api-access-fp769\") pod \"84c5a86f-ba9c-43ee-8faa-4927990018da\" (UID: \"84c5a86f-ba9c-43ee-8faa-4927990018da\") "
Nov 02 12:42:34.189132 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.189097    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-openshift-ca" pod "84c5a86f-ba9c-43ee-8faa-4927990018da" (UID: "84c5a86f-ba9c-43ee-8faa-4927990018da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:42:34.189472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.189452    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:34.191383 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.191359    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-xks-ca" pod "84c5a86f-ba9c-43ee-8faa-4927990018da" (UID: "84c5a86f-ba9c-43ee-8faa-4927990018da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:42:34.202173 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.202127    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c5a86f-ba9c-43ee-8faa-4927990018da-kube-api-access-fp769" pod "84c5a86f-ba9c-43ee-8faa-4927990018da" (UID: "84c5a86f-ba9c-43ee-8faa-4927990018da"). InnerVolumeSpecName "kube-api-access-fp769". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:42:34.253098 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.253066    5879 scope.go:122] "RemoveContainer" containerID="db1536a189ef6ff6de542ba8ba52178fdc9f72052b919b31c7d4990121ebc230"
Nov 02 12:42:34.291205 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.291172    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp769\" (UniqueName: \"kubernetes.io/projected/84c5a86f-ba9c-43ee-8faa-4927990018da-kube-api-access-fp769\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:34.291205 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.291208    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84c5a86f-ba9c-43ee-8faa-4927990018da-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:34.670598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.670254    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:34.835048 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.826042    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:34.863033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.862518    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468" event={"ID":"84c5a86f-ba9c-43ee-8faa-4927990018da","Type":"ContainerDied","Data":"a40a8eaab8b59d19271555c355bded148774c4493419d9a4ee14d50cd6bddf12"}
Nov 02 12:42:34.863033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.862566    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a40a8eaab8b59d19271555c355bded148774c4493419d9a4ee14d50cd6bddf12"
Nov 02 12:42:34.863033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.862621    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"
Nov 02 12:42:34.979120 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:34.979020    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:36.430845 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:36.430813    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:37.047104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:37.047066    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"]
Nov 02 12:42:37.060205 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:37.060170    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e781860f-9hpzs"]
Nov 02 12:42:37.941877 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:37.941837    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:38.017808 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:38.017778    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5efe6021-b363-4f36-bacb-0cb4ac8c0b47" path="/var/lib/kubelet/pods/5efe6021-b363-4f36-bacb-0cb4ac8c0b47/volumes"
Nov 02 12:42:38.073390 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:38.073351    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:39.760443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:39.760399    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:39.980837 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:39.980803    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"]
Nov 02 12:42:39.981523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:39.981504    5879 cpu_manager.go:414] "containerMap: removing container" podUID="84c5a86f-ba9c-43ee-8faa-4927990018da" containerName="revoke-keys"
Nov 02 12:42:39.981523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:39.981523    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c5a86f-ba9c-43ee-8faa-4927990018da" containerName="revoke-keys"
Nov 02 12:42:39.981745 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:39.981732    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c5a86f-ba9c-43ee-8faa-4927990018da" containerName="revoke-keys"
Nov 02 12:42:39.987661 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:39.987635    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:40.004569 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.004539    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"]
Nov 02 12:42:40.013285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.013205    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:42:40.013465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.013423    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:42:40.083133 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.083100    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ac5c135-tmq5j\" (UID: \"1568d745-66f3-49ee-ba2a-2d40ef44f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:40.083307 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.083153    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvv9d\" (UniqueName: \"kubernetes.io/projected/1568d745-66f3-49ee-ba2a-2d40ef44f708-kube-api-access-rvv9d\") pod \"maas-api-revoke-sub-3ac5c135-tmq5j\" (UID: \"1568d745-66f3-49ee-ba2a-2d40ef44f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:40.083379 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.083352    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ac5c135-tmq5j\" (UID: \"1568d745-66f3-49ee-ba2a-2d40ef44f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:40.184368 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.184323    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ac5c135-tmq5j\" (UID: \"1568d745-66f3-49ee-ba2a-2d40ef44f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:40.184559 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.184406    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvv9d\" (UniqueName: \"kubernetes.io/projected/1568d745-66f3-49ee-ba2a-2d40ef44f708-kube-api-access-rvv9d\") pod \"maas-api-revoke-sub-3ac5c135-tmq5j\" (UID: \"1568d745-66f3-49ee-ba2a-2d40ef44f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:40.184906 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.184884    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ac5c135-tmq5j\" (UID: \"1568d745-66f3-49ee-ba2a-2d40ef44f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:40.185044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.185018    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ac5c135-tmq5j\" (UID: \"1568d745-66f3-49ee-ba2a-2d40ef44f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:40.187277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.187257    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ac5c135-tmq5j\" (UID: \"1568d745-66f3-49ee-ba2a-2d40ef44f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:40.209794 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.209765    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvv9d\" (UniqueName: \"kubernetes.io/projected/1568d745-66f3-49ee-ba2a-2d40ef44f708-kube-api-access-rvv9d\") pod \"maas-api-revoke-sub-3ac5c135-tmq5j\" (UID: \"1568d745-66f3-49ee-ba2a-2d40ef44f708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:40.302862 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.302774    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:40.607954 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:42:40.607921    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1568d745_66f3_49ee_ba2a_2d40ef44f708.slice/crio-2bb5e4aea589a318de43995c871b2be0d06262a291698c6cc3990f73aff6b255 WatchSource:0}: Error finding container 2bb5e4aea589a318de43995c871b2be0d06262a291698c6cc3990f73aff6b255: Status 404 returned error can't find the container with id 2bb5e4aea589a318de43995c871b2be0d06262a291698c6cc3990f73aff6b255
Nov 02 12:42:40.609160 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.609136    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"]
Nov 02 12:42:40.918098 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.918062    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j" event={"ID":"1568d745-66f3-49ee-ba2a-2d40ef44f708","Type":"ContainerStarted","Data":"2c21653c7cd9e47e041df24f5be124383c8e0e290f01061643309425aebd88df"}
Nov 02 12:42:40.918546 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.918128    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j" event={"ID":"1568d745-66f3-49ee-ba2a-2d40ef44f708","Type":"ContainerStarted","Data":"2bb5e4aea589a318de43995c871b2be0d06262a291698c6cc3990f73aff6b255"}
Nov 02 12:42:40.963747 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:40.963691    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j" podStartSLOduration=1.9636783580000001 podStartE2EDuration="1.963678358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:40.960891279 +0000 UTC m=+907.067594130" watchObservedRunningTime="2026-11-02 12:42:40.963678358 +0000 UTC m=+907.070381208"
Nov 02 12:42:41.354044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:41.353946    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:41.928204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:41.928171    5879 generic.go:356] "Generic (PLEG): container finished" podID="1568d745-66f3-49ee-ba2a-2d40ef44f708" containerID="2c21653c7cd9e47e041df24f5be124383c8e0e290f01061643309425aebd88df" exitCode=0
Nov 02 12:42:41.928617 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:41.928253    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j" event={"ID":"1568d745-66f3-49ee-ba2a-2d40ef44f708","Type":"ContainerDied","Data":"2c21653c7cd9e47e041df24f5be124383c8e0e290f01061643309425aebd88df"}
Nov 02 12:42:42.461423 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:42.461388    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:42.921151 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:42.921116    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:43.088078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.079289    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:43.235436 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.235407    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:43.258046 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.258012    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-openshift-ca\") pod \"1568d745-66f3-49ee-ba2a-2d40ef44f708\" (UID: \"1568d745-66f3-49ee-ba2a-2d40ef44f708\") "
Nov 02 12:42:43.258277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.258192    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-xks-ca\") pod \"1568d745-66f3-49ee-ba2a-2d40ef44f708\" (UID: \"1568d745-66f3-49ee-ba2a-2d40ef44f708\") "
Nov 02 12:42:43.258352 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.258283    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1568d745-66f3-49ee-ba2a-2d40ef44f708-kube-api-access-rvv9d\" (UniqueName: \"kubernetes.io/projected/1568d745-66f3-49ee-ba2a-2d40ef44f708-kube-api-access-rvv9d\") pod \"1568d745-66f3-49ee-ba2a-2d40ef44f708\" (UID: \"1568d745-66f3-49ee-ba2a-2d40ef44f708\") "
Nov 02 12:42:43.258774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.258743    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-openshift-ca" pod "1568d745-66f3-49ee-ba2a-2d40ef44f708" (UID: "1568d745-66f3-49ee-ba2a-2d40ef44f708"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:42:43.259372 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.259348    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:43.276525 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.276487    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-xks-ca" pod "1568d745-66f3-49ee-ba2a-2d40ef44f708" (UID: "1568d745-66f3-49ee-ba2a-2d40ef44f708"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:42:43.282227 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.282194    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1568d745-66f3-49ee-ba2a-2d40ef44f708-kube-api-access-rvv9d" pod "1568d745-66f3-49ee-ba2a-2d40ef44f708" (UID: "1568d745-66f3-49ee-ba2a-2d40ef44f708"). InnerVolumeSpecName "kube-api-access-rvv9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:42:43.361312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.361266    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvv9d\" (UniqueName: \"kubernetes.io/projected/1568d745-66f3-49ee-ba2a-2d40ef44f708-kube-api-access-rvv9d\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:43.361312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.361307    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1568d745-66f3-49ee-ba2a-2d40ef44f708-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:43.468732 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.468594    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:43.947622 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.947590    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"
Nov 02 12:42:43.947622 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.947597    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j" event={"ID":"1568d745-66f3-49ee-ba2a-2d40ef44f708","Type":"ContainerDied","Data":"2bb5e4aea589a318de43995c871b2be0d06262a291698c6cc3990f73aff6b255"}
Nov 02 12:42:43.947883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:43.947642    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bb5e4aea589a318de43995c871b2be0d06262a291698c6cc3990f73aff6b255"
Nov 02 12:42:44.696525 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:44.696484    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:44.832182 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:42:44.832145    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:42:44.935164 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:44.935130    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"]
Nov 02 12:42:44.935739 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:44.935725    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1568d745-66f3-49ee-ba2a-2d40ef44f708" containerName="revoke-keys"
Nov 02 12:42:44.935791 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:44.935742    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1568d745-66f3-49ee-ba2a-2d40ef44f708" containerName="revoke-keys"
Nov 02 12:42:44.935877 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:44.935868    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1568d745-66f3-49ee-ba2a-2d40ef44f708" containerName="revoke-keys"
Nov 02 12:42:44.938484 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:44.938466    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:44.957714 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:44.957634    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"]
Nov 02 12:42:44.966970 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:44.966941    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:42:44.967210 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:44.967060    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:42:44.996631 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:44.996596    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqnx\" (UniqueName: \"kubernetes.io/projected/cd718141-a4dd-49c3-b6e2-80a95e547bef-kube-api-access-gkqnx\") pod \"maas-api-revoke-sub-21c4c124-4mfgq\" (UID: \"cd718141-a4dd-49c3-b6e2-80a95e547bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:44.996801 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:44.996659    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21c4c124-4mfgq\" (UID: \"cd718141-a4dd-49c3-b6e2-80a95e547bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:44.996842 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:44.996800    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21c4c124-4mfgq\" (UID: \"cd718141-a4dd-49c3-b6e2-80a95e547bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:45.098538 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:45.098490    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqnx\" (UniqueName: \"kubernetes.io/projected/cd718141-a4dd-49c3-b6e2-80a95e547bef-kube-api-access-gkqnx\") pod \"maas-api-revoke-sub-21c4c124-4mfgq\" (UID: \"cd718141-a4dd-49c3-b6e2-80a95e547bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:45.098767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:45.098686    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21c4c124-4mfgq\" (UID: \"cd718141-a4dd-49c3-b6e2-80a95e547bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:45.098836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:45.098792    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21c4c124-4mfgq\" (UID: \"cd718141-a4dd-49c3-b6e2-80a95e547bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:45.099458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:45.099434    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21c4c124-4mfgq\" (UID: \"cd718141-a4dd-49c3-b6e2-80a95e547bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:45.101641 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:45.101615    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21c4c124-4mfgq\" (UID: \"cd718141-a4dd-49c3-b6e2-80a95e547bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:45.124599 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:45.124554    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqnx\" (UniqueName: \"kubernetes.io/projected/cd718141-a4dd-49c3-b6e2-80a95e547bef-kube-api-access-gkqnx\") pod \"maas-api-revoke-sub-21c4c124-4mfgq\" (UID: \"cd718141-a4dd-49c3-b6e2-80a95e547bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:45.251560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:45.251467    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:45.572093 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:42:45.572057    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd718141_a4dd_49c3_b6e2_80a95e547bef.slice/crio-c26f0d5fb9719b0156db1b53aace0a4377af914edcf1494689a595207e03496f WatchSource:0}: Error finding container c26f0d5fb9719b0156db1b53aace0a4377af914edcf1494689a595207e03496f: Status 404 returned error can't find the container with id c26f0d5fb9719b0156db1b53aace0a4377af914edcf1494689a595207e03496f
Nov 02 12:42:45.573683 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:45.573647    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"]
Nov 02 12:42:45.963439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:45.963407    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq" event={"ID":"cd718141-a4dd-49c3-b6e2-80a95e547bef","Type":"ContainerStarted","Data":"c26f0d5fb9719b0156db1b53aace0a4377af914edcf1494689a595207e03496f"}
Nov 02 12:42:46.165345 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.165310    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"]
Nov 02 12:42:46.169057 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.169035    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.198029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.197968    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"]
Nov 02 12:42:46.218045 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.217946    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.218294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.218259    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.218406 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.218324    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.218467 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.218407    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.218467 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.218456    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.218676 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.218559    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6vq9\" (UniqueName: \"kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-kube-api-access-t6vq9\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.218676 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.218585    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.218676 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.218617    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.218896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.218742    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.320331 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.320301    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.320331 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.320338    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.320531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.320367    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.320531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.320398    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.320531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.320435    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.320675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.320591    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6vq9\" (UniqueName: \"kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-kube-api-access-t6vq9\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.320675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.320638    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.320784 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.320688    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.320784 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.320700    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.320784 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.320751    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.320938 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.320796    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.321077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.321050    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.321210 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.321186    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.321354 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.321334    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.322939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.322918    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.323061 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.322948    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.345525 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.345497    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.345728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.345705    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6vq9\" (UniqueName: \"kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-kube-api-access-t6vq9\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.482689 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.482607    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:46.537226 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.537189    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:46.675485 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.675420    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:46.807236 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:42:46.807175    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3b2efe_fbb1_4c96_abf6_d9bd3b31ebb8.slice/crio-9746f385e3aebfc6163158ee871fe64912e74b3ea41b974beb94c599b387b5a6 WatchSource:0}: Error finding container 9746f385e3aebfc6163158ee871fe64912e74b3ea41b974beb94c599b387b5a6: Status 404 returned error can't find the container with id 9746f385e3aebfc6163158ee871fe64912e74b3ea41b974beb94c599b387b5a6
Nov 02 12:42:46.808833 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.808806    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"]
Nov 02 12:42:46.809903 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.809868    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:42:46.810056 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.809943    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:42:46.810056 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.809984    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:42:46.812920 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.812896    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:46.973101 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.973067    5879 generic.go:356] "Generic (PLEG): container finished" podID="cd718141-a4dd-49c3-b6e2-80a95e547bef" containerID="62e9e8b626eff29ad2e9a44f9119c97285c120620390267ba86b84a7865d1925" exitCode=0
Nov 02 12:42:46.973565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.973162    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq" event={"ID":"cd718141-a4dd-49c3-b6e2-80a95e547bef","Type":"ContainerDied","Data":"62e9e8b626eff29ad2e9a44f9119c97285c120620390267ba86b84a7865d1925"}
Nov 02 12:42:46.975252 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.975224    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" event={"ID":"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8","Type":"ContainerStarted","Data":"8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c"}
Nov 02 12:42:46.975414 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:46.975274    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" event={"ID":"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8","Type":"ContainerStarted","Data":"9746f385e3aebfc6163158ee871fe64912e74b3ea41b974beb94c599b387b5a6"}
Nov 02 12:42:47.085539 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:47.085421    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" podStartSLOduration=1.085406377 podStartE2EDuration="1.085406377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:42:47.084358087 +0000 UTC m=+913.191060939" watchObservedRunningTime="2026-11-02 12:42:47.085406377 +0000 UTC m=+913.192109227"
Nov 02 12:42:47.483243 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:47.483205    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:47.485238 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:47.485210    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:42:47.485388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:47.485264    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 12:42:48.219778 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.219754    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:48.352180 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.352067    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd718141-a4dd-49c3-b6e2-80a95e547bef-kube-api-access-gkqnx\" (UniqueName: \"kubernetes.io/projected/cd718141-a4dd-49c3-b6e2-80a95e547bef-kube-api-access-gkqnx\") pod \"cd718141-a4dd-49c3-b6e2-80a95e547bef\" (UID: \"cd718141-a4dd-49c3-b6e2-80a95e547bef\") "
Nov 02 12:42:48.352348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.352183    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-openshift-ca\") pod \"cd718141-a4dd-49c3-b6e2-80a95e547bef\" (UID: \"cd718141-a4dd-49c3-b6e2-80a95e547bef\") "
Nov 02 12:42:48.352455 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.352436    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-xks-ca\") pod \"cd718141-a4dd-49c3-b6e2-80a95e547bef\" (UID: \"cd718141-a4dd-49c3-b6e2-80a95e547bef\") "
Nov 02 12:42:48.353738 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.353442    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-openshift-ca" pod "cd718141-a4dd-49c3-b6e2-80a95e547bef" (UID: "cd718141-a4dd-49c3-b6e2-80a95e547bef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:42:48.353738 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.353682    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:48.359278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.359246    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd718141-a4dd-49c3-b6e2-80a95e547bef-kube-api-access-gkqnx" pod "cd718141-a4dd-49c3-b6e2-80a95e547bef" (UID: "cd718141-a4dd-49c3-b6e2-80a95e547bef"). InnerVolumeSpecName "kube-api-access-gkqnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:42:48.359616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.359589    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-xks-ca" pod "cd718141-a4dd-49c3-b6e2-80a95e547bef" (UID: "cd718141-a4dd-49c3-b6e2-80a95e547bef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:42:48.394243 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.394104    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:48.455542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.455505    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkqnx\" (UniqueName: \"kubernetes.io/projected/cd718141-a4dd-49c3-b6e2-80a95e547bef-kube-api-access-gkqnx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:48.455542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.455536    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd718141-a4dd-49c3-b6e2-80a95e547bef-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:42:48.507522 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.507486    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:42:48.507793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.507548    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 12:42:48.543290 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.543259    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:48.735642 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:48.735553    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:49.016668 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:49.016592    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"
Nov 02 12:42:49.016668 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:49.016650    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq" event={"ID":"cd718141-a4dd-49c3-b6e2-80a95e547bef","Type":"ContainerDied","Data":"c26f0d5fb9719b0156db1b53aace0a4377af914edcf1494689a595207e03496f"}
Nov 02 12:42:49.016846 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:49.016687    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c26f0d5fb9719b0156db1b53aace0a4377af914edcf1494689a595207e03496f"
Nov 02 12:42:49.500020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:49.491443    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:42:49.500020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:49.491523    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 12:42:50.260814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:50.260773    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:50.414020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:50.413480    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:50.493019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:50.489434    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:42:50.493019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:50.489507    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 12:42:50.812741 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:50.812709    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:51.440752 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:51.440718    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:51.485032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:51.483723    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:42:51.485032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:51.483798    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 12:42:51.999519 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:51.999480    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:52.047728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:52.047667    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"]
Nov 02 12:42:52.055576 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:52.055550    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4606331a-zj89d"]
Nov 02 12:42:52.205863 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:52.205820    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:52.492366 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:52.492329    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:42:52.496042 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:52.495978    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 12:42:53.315788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:53.315746    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:53.483480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:53.483434    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 12:42:53.483682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:53.483508    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 12:42:53.962211 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:53.962180    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:54.023553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:54.023506    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33290e89-7f9a-4899-9c57-8b4231b2dd5f" path="/var/lib/kubelet/pods/33290e89-7f9a-4899-9c57-8b4231b2dd5f/volumes"
Nov 02 12:42:54.685667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:54.685639    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:54.686105 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:54.685984    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:54.686883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:54.686864    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 12:42:55.322554 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:55.322512    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:55.484570 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:55.484536    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:56.357677 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:56.357603    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:57.353504 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:57.353461    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:57.503773 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:57.503745    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:58.354019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:58.353950    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:59.726566 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:59.726534    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:42:59.853388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:42:59.853352    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:00.082423 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:00.082092    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"]
Nov 02 12:43:00.114075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:00.114038    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"]
Nov 02 12:43:00.140019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:00.132950    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7283989-l8psw"]
Nov 02 12:43:00.140019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:00.139051    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbacaaa9-cjrdh"]
Nov 02 12:43:00.226269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:00.226216    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:00.401314 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:00.401279    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:01.931727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:01.931694    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:02.020433 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:02.020391    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d031a40-16b1-4f22-8a9f-216404661f78" path="/var/lib/kubelet/pods/2d031a40-16b1-4f22-8a9f-216404661f78/volumes"
Nov 02 12:43:02.021105 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:02.021080    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0c96d2-31a7-44a6-b3ec-7544395a47aa" path="/var/lib/kubelet/pods/7f0c96d2-31a7-44a6-b3ec-7544395a47aa/volumes"
Nov 02 12:43:02.224534 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:02.224452    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:03.032935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:03.032903    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"]
Nov 02 12:43:03.049243 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:03.049208    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39a9d2ca-r7ztn"]
Nov 02 12:43:03.518034 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:03.517987    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:03.656729 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:03.656694    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:03.877454 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:03.877424    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:04.016795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:04.016769    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8175b87e-e393-44de-9a73-66b76e696d50" path="/var/lib/kubelet/pods/8175b87e-e393-44de-9a73-66b76e696d50/volumes"
Nov 02 12:43:05.247481 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:05.247448    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:06.393189 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:06.393153    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:06.848513 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:06.848431    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:08.187802 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:08.187750    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:08.858160 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:08.858119    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:10.467451 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:10.467409    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:12.101108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:12.101076    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:12.244568 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:12.244535    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:12.730792 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:12.730756    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:12.858819 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:12.858786    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:13.456253 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:13.456206    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:13.839100 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:13.838959    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:13.974732 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:13.974697    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:14.697199 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:14.697167    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:14.821495 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:43:14.821462    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:43:15.032622 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:15.032540    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"]
Nov 02 12:43:15.045954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:15.045926    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1b05aaf-jf8x9"]
Nov 02 12:43:15.556697 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:15.556664    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:16.017525 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:16.017492    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef3de9e5-ab1d-464d-811b-3f451ac760c6" path="/var/lib/kubelet/pods/ef3de9e5-ab1d-464d-811b-3f451ac760c6/volumes"
Nov 02 12:43:17.266602 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:17.266566    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:18.032332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:18.032300    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"]
Nov 02 12:43:18.046270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:18.046236    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56bc10a0-64cww"]
Nov 02 12:43:18.846185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:18.846151    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:20.031829 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:20.031787    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a194bf5-efe2-44b4-b546-6251943cc662" path="/var/lib/kubelet/pods/3a194bf5-efe2-44b4-b546-6251943cc662/volumes"
Nov 02 12:43:20.282094 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:20.274390    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:20.478019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:20.464519    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:21.200967 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:21.200935    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:21.766747 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:21.766705    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:22.916502 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:22.916455    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:23.135004 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:23.134957    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:24.959022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:24.956971    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:26.757380 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:26.757342    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:27.448231 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:27.448197    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:28.042154 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.042117    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:28.043945 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.043916    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"]
Nov 02 12:43:28.053579 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.053550    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8395a3b-ndnlv"]
Nov 02 12:43:28.064152 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.064115    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"]
Nov 02 12:43:28.064793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.064774    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cd718141-a4dd-49c3-b6e2-80a95e547bef" containerName="revoke-keys"
Nov 02 12:43:28.064793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.064791    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd718141-a4dd-49c3-b6e2-80a95e547bef" containerName="revoke-keys"
Nov 02 12:43:28.064952 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.064887    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd718141-a4dd-49c3-b6e2-80a95e547bef" containerName="revoke-keys"
Nov 02 12:43:28.067929 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.067909    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.102025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.101974    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"]
Nov 02 12:43:28.165294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.165262    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9f7f01-openshift-default-dockercfg-hkzth\""
Nov 02 12:43:28.176008 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.175947    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-certs\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.176159 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.176104    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-token\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.176227 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.176158    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a8dae1-0b29-4f71-9603-9793e5f50e28-istiod-ca-cert\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.176227 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.176192    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msvt4\" (UniqueName: \"kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-kube-api-access-msvt4\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.176343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.176325    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-data\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.176390 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.176380    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.176446 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.176399    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-podinfo\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.176556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.176535    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-credential-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.176616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.176581    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-envoy\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.193241 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.193209    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:28.278104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.278066    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-credential-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.278104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.278107    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-envoy\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.278427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.278146    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-certs\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.278494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.278476    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-token\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.278550 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.278519    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-credential-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.278596 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.278577    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a8dae1-0b29-4f71-9603-9793e5f50e28-istiod-ca-cert\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.278649 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.278615    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-certs\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.278649 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.278622    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msvt4\" (UniqueName: \"kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-kube-api-access-msvt4\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.278753 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.278723    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-data\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.278811 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.278787    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.278883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.278812    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-podinfo\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.279143 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.279114    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-data\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.279269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.279181    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-socket\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.279396 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.279377    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a8dae1-0b29-4f71-9603-9793e5f50e28-istiod-ca-cert\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.280675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.280655    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-envoy\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.281672 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.281647    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-podinfo\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.305855 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.305769    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-token\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.306019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.305911    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msvt4\" (UniqueName: \"kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-kube-api-access-msvt4\") pod \"e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") " pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.384861 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.384826    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:28.393201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.393166    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:28.747425 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:43:28.747387    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a8dae1_0b29_4f71_9603_9793e5f50e28.slice/crio-e1c93cc6fc082b2a6b4cdd0a0d3cf08aff170a7bc4d6137db9f5438016ba5d82 WatchSource:0}: Error finding container e1c93cc6fc082b2a6b4cdd0a0d3cf08aff170a7bc4d6137db9f5438016ba5d82: Status 404 returned error can't find the container with id e1c93cc6fc082b2a6b4cdd0a0d3cf08aff170a7bc4d6137db9f5438016ba5d82
Nov 02 12:43:28.749354 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.749329    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"]
Nov 02 12:43:28.750736 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.750697    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:43:28.750843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.750781    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:43:28.750843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:28.750825    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:43:29.250050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:29.248872    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:29.488400 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:29.488364    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:29.618021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:29.616718    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l" event={"ID":"62a8dae1-0b29-4f71-9603-9793e5f50e28","Type":"ContainerStarted","Data":"7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef"}
Nov 02 12:43:29.618021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:29.616792    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l" event={"ID":"62a8dae1-0b29-4f71-9603-9793e5f50e28","Type":"ContainerStarted","Data":"e1c93cc6fc082b2a6b4cdd0a0d3cf08aff170a7bc4d6137db9f5438016ba5d82"}
Nov 02 12:43:29.699019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:29.688283    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l" podStartSLOduration=1.688261996 podStartE2EDuration="1.688261996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:43:29.68661746 +0000 UTC m=+955.793320316" watchObservedRunningTime="2026-11-02 12:43:29.688261996 +0000 UTC m=+955.794964848"
Nov 02 12:43:30.023365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:30.023273    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961ca2b3-89b7-474f-a146-bab598bc9d39" path="/var/lib/kubelet/pods/961ca2b3-89b7-474f-a146-bab598bc9d39/volumes"
Nov 02 12:43:30.209016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:30.203052    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:30.393019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:30.387372    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:30.405175 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:30.405132    5879 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 12:43:30.405382 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:30.405215    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l" podUID="62a8dae1-0b29-4f71-9603-9793e5f50e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 12:43:31.386021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:31.385400    5879 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 12:43:31.386021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:31.385464    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l" podUID="62a8dae1-0b29-4f71-9603-9793e5f50e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 12:43:32.061344 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:32.061315    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:32.406474 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:32.406435    5879 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 12:43:32.406657 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:32.406517    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l" podUID="62a8dae1-0b29-4f71-9603-9793e5f50e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 12:43:33.393453 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:33.393416    5879 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 12:43:33.394416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:33.394387    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l" podUID="62a8dae1-0b29-4f71-9603-9793e5f50e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 12:43:33.509499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:33.509454    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:33.700495 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:33.700402    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:33.915547 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:33.915502    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:34.308944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:34.308909    5879 scope.go:122] "RemoveContainer" containerID="1d0de91759bd46fb302012294bb96b8e6f9f452e838d43748cc6425fdcfe28dd"
Nov 02 12:43:34.407525 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:34.407483    5879 patch_prober.go:29] interesting pod/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body=
Nov 02 12:43:34.408080 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:34.407583    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l" podUID="62a8dae1-0b29-4f71-9603-9793e5f50e28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused"
Nov 02 12:43:34.829353 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:34.829311    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:35.302771 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:35.302735    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:35.396153 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:35.396124    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:35.397335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:35.397309    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:35.398408 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:35.398384    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:43:35.564955 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:35.564863    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:36.053618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:36.053557    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:36.229709 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:36.229646    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:36.748426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:36.748391    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:37.203488 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.203445    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"]
Nov 02 12:43:37.208667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.208638    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:37.233241 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.233201    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"]
Nov 02 12:43:37.271365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.271321    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:37.284827 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.284783    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:43:37.285051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.284885    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:43:37.317956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.317917    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b\" (UID: \"6e3fbe50-f114-4e40-9e98-097a31d2251b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:37.318174 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.318047    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b\" (UID: \"6e3fbe50-f114-4e40-9e98-097a31d2251b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:37.318406 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.318374    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9g42\" (UniqueName: \"kubernetes.io/projected/6e3fbe50-f114-4e40-9e98-097a31d2251b-kube-api-access-c9g42\") pod \"maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b\" (UID: \"6e3fbe50-f114-4e40-9e98-097a31d2251b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:37.419656 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.419620    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b\" (UID: \"6e3fbe50-f114-4e40-9e98-097a31d2251b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:37.419846 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.419710    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b\" (UID: \"6e3fbe50-f114-4e40-9e98-097a31d2251b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:37.419906 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.419867    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9g42\" (UniqueName: \"kubernetes.io/projected/6e3fbe50-f114-4e40-9e98-097a31d2251b-kube-api-access-c9g42\") pod \"maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b\" (UID: \"6e3fbe50-f114-4e40-9e98-097a31d2251b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:37.420486 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.420461    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b\" (UID: \"6e3fbe50-f114-4e40-9e98-097a31d2251b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:37.422271 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.422250    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b\" (UID: \"6e3fbe50-f114-4e40-9e98-097a31d2251b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:37.473769 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.473693    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9g42\" (UniqueName: \"kubernetes.io/projected/6e3fbe50-f114-4e40-9e98-097a31d2251b-kube-api-access-c9g42\") pod \"maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b\" (UID: \"6e3fbe50-f114-4e40-9e98-097a31d2251b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:37.506337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.506293    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:37.535285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.535244    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:37.872865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.872836    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"]
Nov 02 12:43:37.935352 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:37.931671    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:38.831142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:38.831111    5879 generic.go:356] "Generic (PLEG): container finished" podID="6e3fbe50-f114-4e40-9e98-097a31d2251b" containerID="0c9969bda84a3469b2a9094fd937ae844d37a72dfae01adac6b0e0c0f9295410" exitCode=0
Nov 02 12:43:38.831319 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:38.831191    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b" event={"ID":"6e3fbe50-f114-4e40-9e98-097a31d2251b","Type":"ContainerDied","Data":"0c9969bda84a3469b2a9094fd937ae844d37a72dfae01adac6b0e0c0f9295410"}
Nov 02 12:43:38.831319 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:38.831243    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b" event={"ID":"6e3fbe50-f114-4e40-9e98-097a31d2251b","Type":"ContainerStarted","Data":"ff62c3df03a0efcc585432d40ad529ba81e164dff8776c7f0ab2cee6c60fd745"}
Nov 02 12:43:39.895944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:39.895909    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:40.014765 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.014740    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:40.098305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.097943    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-openshift-ca\") pod \"6e3fbe50-f114-4e40-9e98-097a31d2251b\" (UID: \"6e3fbe50-f114-4e40-9e98-097a31d2251b\") "
Nov 02 12:43:40.098305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.098139    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3fbe50-f114-4e40-9e98-097a31d2251b-kube-api-access-c9g42\" (UniqueName: \"kubernetes.io/projected/6e3fbe50-f114-4e40-9e98-097a31d2251b-kube-api-access-c9g42\") pod \"6e3fbe50-f114-4e40-9e98-097a31d2251b\" (UID: \"6e3fbe50-f114-4e40-9e98-097a31d2251b\") "
Nov 02 12:43:40.098305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.098211    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-xks-ca\") pod \"6e3fbe50-f114-4e40-9e98-097a31d2251b\" (UID: \"6e3fbe50-f114-4e40-9e98-097a31d2251b\") "
Nov 02 12:43:40.098596 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.098442    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-openshift-ca" pod "6e3fbe50-f114-4e40-9e98-097a31d2251b" (UID: "6e3fbe50-f114-4e40-9e98-097a31d2251b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:43:40.098916 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.098889    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:43:40.100898 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.100874    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3fbe50-f114-4e40-9e98-097a31d2251b-kube-api-access-c9g42" pod "6e3fbe50-f114-4e40-9e98-097a31d2251b" (UID: "6e3fbe50-f114-4e40-9e98-097a31d2251b"). InnerVolumeSpecName "kube-api-access-c9g42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:43:40.101445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.101420    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-xks-ca" pod "6e3fbe50-f114-4e40-9e98-097a31d2251b" (UID: "6e3fbe50-f114-4e40-9e98-097a31d2251b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:43:40.199727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.199695    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9g42\" (UniqueName: \"kubernetes.io/projected/6e3fbe50-f114-4e40-9e98-097a31d2251b-kube-api-access-c9g42\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:43:40.199727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.199722    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e3fbe50-f114-4e40-9e98-097a31d2251b-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:43:40.847728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.847697    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"
Nov 02 12:43:40.847910 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.847730    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b" event={"ID":"6e3fbe50-f114-4e40-9e98-097a31d2251b","Type":"ContainerDied","Data":"ff62c3df03a0efcc585432d40ad529ba81e164dff8776c7f0ab2cee6c60fd745"}
Nov 02 12:43:40.847910 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:40.847802    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff62c3df03a0efcc585432d40ad529ba81e164dff8776c7f0ab2cee6c60fd745"
Nov 02 12:43:41.755189 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:41.755154    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:41.944093 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:41.944049    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:43.877257 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:43.877222    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:44.022163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:44.022123    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:44.163306 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:44.163223    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:44.822676 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:43:44.822642    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:43:45.637286 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:45.637248    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:45.851865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:45.851823    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:45.981641 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:45.981552    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:46.481216 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:46.481184    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:46.979292 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:46.979254    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:47.498479 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:47.498447    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:47.970807 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:47.970772    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:48.106165 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:48.106132    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:49.147658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:49.147628    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:50.678229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:50.678190    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:50.826648 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:50.826616    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:52.385962 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:52.381186    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:54.168593 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:54.168528    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:55.842067 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:55.842024    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:56.033764 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:56.033729    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"]
Nov 02 12:43:56.047144 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:56.047113    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f7985cb-55qfg"]
Nov 02 12:43:57.368334 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:57.368302    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:58.017541 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:58.017512    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4c9c358-856a-4200-8cc3-80a379e3fe14" path="/var/lib/kubelet/pods/b4c9c358-856a-4200-8cc3-80a379e3fe14/volumes"
Nov 02 12:43:58.897673 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:58.897620    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:59.044757 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:59.044724    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:43:59.220177 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:43:59.220097    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:00.657302 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:00.657263    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:02.352320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:02.352271    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:03.789411 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.789379    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"]
Nov 02 12:44:03.789987 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.789974    5879 cpu_manager.go:414] "containerMap: removing container" podUID="6e3fbe50-f114-4e40-9e98-097a31d2251b" containerName="revoke-keys"
Nov 02 12:44:03.790050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.790003    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3fbe50-f114-4e40-9e98-097a31d2251b" containerName="revoke-keys"
Nov 02 12:44:03.790183 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.790172    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3fbe50-f114-4e40-9e98-097a31d2251b" containerName="revoke-keys"
Nov 02 12:44:03.794229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.794207    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:03.814827 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.814792    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"]
Nov 02 12:44:03.823208 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.823163    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:44:03.823369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.823252    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:44:03.889926 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.889890    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b610da10-54dxc\" (UID: \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:03.890145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.890115    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5g4x\" (UniqueName: \"kubernetes.io/projected/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-kube-api-access-z5g4x\") pod \"maas-api-revoke-sub-b610da10-54dxc\" (UID: \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:03.890392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.890376    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b610da10-54dxc\" (UID: \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:03.992086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.992045    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b610da10-54dxc\" (UID: \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:03.992287 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.992097    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b610da10-54dxc\" (UID: \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:03.992287 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.992178    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5g4x\" (UniqueName: \"kubernetes.io/projected/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-kube-api-access-z5g4x\") pod \"maas-api-revoke-sub-b610da10-54dxc\" (UID: \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:03.992745 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.992718    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b610da10-54dxc\" (UID: \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:03.994620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:03.994591    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b610da10-54dxc\" (UID: \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:04.017496 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:04.017466    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5g4x\" (UniqueName: \"kubernetes.io/projected/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-kube-api-access-z5g4x\") pod \"maas-api-revoke-sub-b610da10-54dxc\" (UID: \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:04.108671 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:04.108583    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:04.207343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:04.207311    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:04.416080 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:44:04.416042    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2946fc9e_74f5_40ce_ae78_ac18fbb6a102.slice/crio-2e9dca7d5a1069ae150bcf4a6ba91b53dc138873704cd55e2b713112a703d404 WatchSource:0}: Error finding container 2e9dca7d5a1069ae150bcf4a6ba91b53dc138873704cd55e2b713112a703d404: Status 404 returned error can't find the container with id 2e9dca7d5a1069ae150bcf4a6ba91b53dc138873704cd55e2b713112a703d404
Nov 02 12:44:04.417155 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:04.417127    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"]
Nov 02 12:44:05.093194 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:05.093108    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc" event={"ID":"2946fc9e-74f5-40ce-ae78-ac18fbb6a102","Type":"ContainerStarted","Data":"431c8afe851c1e8e7b2f6a54130ef8ba3c764688e76fd84fc074e98e9a678190"}
Nov 02 12:44:05.093194 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:05.093164    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc" event={"ID":"2946fc9e-74f5-40ce-ae78-ac18fbb6a102","Type":"ContainerStarted","Data":"2e9dca7d5a1069ae150bcf4a6ba91b53dc138873704cd55e2b713112a703d404"}
Nov 02 12:44:05.149556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:05.149497    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc" podStartSLOduration=2.14948269 podStartE2EDuration="2.14948269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:44:05.146367043 +0000 UTC m=+991.253069895" watchObservedRunningTime="2026-11-02 12:44:05.14948269 +0000 UTC m=+991.256185579"
Nov 02 12:44:05.816120 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:05.816032    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:06.101960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:06.101869    5879 generic.go:356] "Generic (PLEG): container finished" podID="2946fc9e-74f5-40ce-ae78-ac18fbb6a102" containerID="431c8afe851c1e8e7b2f6a54130ef8ba3c764688e76fd84fc074e98e9a678190" exitCode=0
Nov 02 12:44:06.102446 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:06.101953    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc" event={"ID":"2946fc9e-74f5-40ce-ae78-ac18fbb6a102","Type":"ContainerDied","Data":"431c8afe851c1e8e7b2f6a54130ef8ba3c764688e76fd84fc074e98e9a678190"}
Nov 02 12:44:07.276899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:07.276876    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:07.451137 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:07.451099    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-kube-api-access-z5g4x\" (UniqueName: \"kubernetes.io/projected/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-kube-api-access-z5g4x\") pod \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\" (UID: \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\") "
Nov 02 12:44:07.451394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:07.451150    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-openshift-ca\") pod \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\" (UID: \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\") "
Nov 02 12:44:07.451394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:07.451265    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-xks-ca\") pod \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\" (UID: \"2946fc9e-74f5-40ce-ae78-ac18fbb6a102\") "
Nov 02 12:44:07.451653 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:07.451616    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-openshift-ca" pod "2946fc9e-74f5-40ce-ae78-ac18fbb6a102" (UID: "2946fc9e-74f5-40ce-ae78-ac18fbb6a102"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:07.451933 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:07.451914    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:07.453510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:07.453483    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-xks-ca" pod "2946fc9e-74f5-40ce-ae78-ac18fbb6a102" (UID: "2946fc9e-74f5-40ce-ae78-ac18fbb6a102"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:44:07.453687 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:07.453667    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-kube-api-access-z5g4x" pod "2946fc9e-74f5-40ce-ae78-ac18fbb6a102" (UID: "2946fc9e-74f5-40ce-ae78-ac18fbb6a102"). InnerVolumeSpecName "kube-api-access-z5g4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:07.553707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:07.553672    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:07.553707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:07.553703    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5g4x\" (UniqueName: \"kubernetes.io/projected/2946fc9e-74f5-40ce-ae78-ac18fbb6a102-kube-api-access-z5g4x\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:07.624882 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:07.624840    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:08.119623 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:08.119594    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"
Nov 02 12:44:08.119836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:08.119627    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc" event={"ID":"2946fc9e-74f5-40ce-ae78-ac18fbb6a102","Type":"ContainerDied","Data":"2e9dca7d5a1069ae150bcf4a6ba91b53dc138873704cd55e2b713112a703d404"}
Nov 02 12:44:08.119836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:08.119682    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e9dca7d5a1069ae150bcf4a6ba91b53dc138873704cd55e2b713112a703d404"
Nov 02 12:44:09.033169 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:09.033135    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"]
Nov 02 12:44:09.048317 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:09.048279    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ba245f-gfb5q"]
Nov 02 12:44:09.287265 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:09.287126    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:09.981481 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:09.981444    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"]
Nov 02 12:44:09.982421 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:09.982401    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2946fc9e-74f5-40ce-ae78-ac18fbb6a102" containerName="revoke-keys"
Nov 02 12:44:09.982482 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:09.982425    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2946fc9e-74f5-40ce-ae78-ac18fbb6a102" containerName="revoke-keys"
Nov 02 12:44:09.982644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:09.982633    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2946fc9e-74f5-40ce-ae78-ac18fbb6a102" containerName="revoke-keys"
Nov 02 12:44:09.987186 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:09.987164    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:10.005421 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.005392    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"]
Nov 02 12:44:10.014374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.014345    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:44:10.014374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.014358    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:44:10.018986 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.018957    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1dcd822-a886-418d-8f63-6b7daa823f92" path="/var/lib/kubelet/pods/e1dcd822-a886-418d-8f63-6b7daa823f92/volumes"
Nov 02 12:44:10.094229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.094192    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f86dbf6-p5d9w\" (UID: \"d11c3ef5-d6b3-45f9-a495-3119c068787f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:10.094633 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.094353    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnrfz\" (UniqueName: \"kubernetes.io/projected/d11c3ef5-d6b3-45f9-a495-3119c068787f-kube-api-access-vnrfz\") pod \"maas-api-revoke-sub-1f86dbf6-p5d9w\" (UID: \"d11c3ef5-d6b3-45f9-a495-3119c068787f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:10.094633 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.094406    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f86dbf6-p5d9w\" (UID: \"d11c3ef5-d6b3-45f9-a495-3119c068787f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:10.196134 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.196093    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnrfz\" (UniqueName: \"kubernetes.io/projected/d11c3ef5-d6b3-45f9-a495-3119c068787f-kube-api-access-vnrfz\") pod \"maas-api-revoke-sub-1f86dbf6-p5d9w\" (UID: \"d11c3ef5-d6b3-45f9-a495-3119c068787f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:10.196343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.196153    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f86dbf6-p5d9w\" (UID: \"d11c3ef5-d6b3-45f9-a495-3119c068787f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:10.196343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.196316    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f86dbf6-p5d9w\" (UID: \"d11c3ef5-d6b3-45f9-a495-3119c068787f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:10.196985 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.196963    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f86dbf6-p5d9w\" (UID: \"d11c3ef5-d6b3-45f9-a495-3119c068787f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:10.198954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.198928    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f86dbf6-p5d9w\" (UID: \"d11c3ef5-d6b3-45f9-a495-3119c068787f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:10.221542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.221506    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnrfz\" (UniqueName: \"kubernetes.io/projected/d11c3ef5-d6b3-45f9-a495-3119c068787f-kube-api-access-vnrfz\") pod \"maas-api-revoke-sub-1f86dbf6-p5d9w\" (UID: \"d11c3ef5-d6b3-45f9-a495-3119c068787f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:10.303732 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.303643    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:10.620209 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:10.620179    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"]
Nov 02 12:44:10.620559 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:44:10.620529    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11c3ef5_d6b3_45f9_a495_3119c068787f.slice/crio-15da2953feb94110bf0bb022675da57d11626352a1a583290e508b277c4bd492 WatchSource:0}: Error finding container 15da2953feb94110bf0bb022675da57d11626352a1a583290e508b277c4bd492: Status 404 returned error can't find the container with id 15da2953feb94110bf0bb022675da57d11626352a1a583290e508b277c4bd492
Nov 02 12:44:11.076056 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:11.076022    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:11.148205 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:11.148125    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w" event={"ID":"d11c3ef5-d6b3-45f9-a495-3119c068787f","Type":"ContainerStarted","Data":"1f9910d663715538f5cbf6a62495fa39b0b1a3f7677c3b061f4fe820d661fc37"}
Nov 02 12:44:11.148205 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:11.148181    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w" event={"ID":"d11c3ef5-d6b3-45f9-a495-3119c068787f","Type":"ContainerStarted","Data":"15da2953feb94110bf0bb022675da57d11626352a1a583290e508b277c4bd492"}
Nov 02 12:44:11.193436 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:11.193378    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w" podStartSLOduration=2.193361462 podStartE2EDuration="2.193361462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:44:11.191108291 +0000 UTC m=+997.297811155" watchObservedRunningTime="2026-11-02 12:44:11.193361462 +0000 UTC m=+997.300064312"
Nov 02 12:44:12.157670 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:12.157633    5879 generic.go:356] "Generic (PLEG): container finished" podID="d11c3ef5-d6b3-45f9-a495-3119c068787f" containerID="1f9910d663715538f5cbf6a62495fa39b0b1a3f7677c3b061f4fe820d661fc37" exitCode=0
Nov 02 12:44:12.158125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:12.157716    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w" event={"ID":"d11c3ef5-d6b3-45f9-a495-3119c068787f","Type":"ContainerDied","Data":"1f9910d663715538f5cbf6a62495fa39b0b1a3f7677c3b061f4fe820d661fc37"}
Nov 02 12:44:12.747795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:12.747756    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:13.360806 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:13.360772    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:13.444689 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:13.444549    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11c3ef5-d6b3-45f9-a495-3119c068787f-kube-api-access-vnrfz\" (UniqueName: \"kubernetes.io/projected/d11c3ef5-d6b3-45f9-a495-3119c068787f-kube-api-access-vnrfz\") pod \"d11c3ef5-d6b3-45f9-a495-3119c068787f\" (UID: \"d11c3ef5-d6b3-45f9-a495-3119c068787f\") "
Nov 02 12:44:13.444689 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:13.444616    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-xks-ca\") pod \"d11c3ef5-d6b3-45f9-a495-3119c068787f\" (UID: \"d11c3ef5-d6b3-45f9-a495-3119c068787f\") "
Nov 02 12:44:13.444689 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:13.444682    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-openshift-ca\") pod \"d11c3ef5-d6b3-45f9-a495-3119c068787f\" (UID: \"d11c3ef5-d6b3-45f9-a495-3119c068787f\") "
Nov 02 12:44:13.445156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:13.445124    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-openshift-ca" pod "d11c3ef5-d6b3-45f9-a495-3119c068787f" (UID: "d11c3ef5-d6b3-45f9-a495-3119c068787f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:13.445457 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:13.445433    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:13.447622 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:13.447593    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11c3ef5-d6b3-45f9-a495-3119c068787f-kube-api-access-vnrfz" pod "d11c3ef5-d6b3-45f9-a495-3119c068787f" (UID: "d11c3ef5-d6b3-45f9-a495-3119c068787f"). InnerVolumeSpecName "kube-api-access-vnrfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:13.447792 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:13.447777    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-xks-ca" pod "d11c3ef5-d6b3-45f9-a495-3119c068787f" (UID: "d11c3ef5-d6b3-45f9-a495-3119c068787f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:44:13.546954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:13.546914    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnrfz\" (UniqueName: \"kubernetes.io/projected/d11c3ef5-d6b3-45f9-a495-3119c068787f-kube-api-access-vnrfz\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:13.546954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:13.546948    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d11c3ef5-d6b3-45f9-a495-3119c068787f-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:14.178051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:14.177983    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"
Nov 02 12:44:14.178229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:14.178043    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w" event={"ID":"d11c3ef5-d6b3-45f9-a495-3119c068787f","Type":"ContainerDied","Data":"15da2953feb94110bf0bb022675da57d11626352a1a583290e508b277c4bd492"}
Nov 02 12:44:14.178229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:14.178103    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15da2953feb94110bf0bb022675da57d11626352a1a583290e508b277c4bd492"
Nov 02 12:44:14.250064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:14.249372    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:14.392004 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:14.391946    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:14.822382 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:44:14.822296    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:44:15.928655 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:15.928620    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:16.049296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:16.049261    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:17.752628 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:17.752594    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:19.000369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:19.000333    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:19.128936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:19.128906    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:19.422180 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:19.422139    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:21.130377 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:21.130336    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:22.207021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:22.205132    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:22.480300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:22.456530    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:22.636034 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:22.632327    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:23.086680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:23.086632    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:23.516348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:23.516258    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:23.664032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:23.663968    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:25.268382 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:25.268348    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:25.823671 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:25.823596    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:25.964679 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:25.964645    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:26.724581 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:26.724546    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:27.304814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:27.303713    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:27.381043 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:27.380986    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"]
Nov 02 12:44:27.395911 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:27.395875    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cd1386cb-5js6b"]
Nov 02 12:44:27.441936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:27.441894    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:28.017404 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:28.017374    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e3fbe50-f114-4e40-9e98-097a31d2251b" path="/var/lib/kubelet/pods/6e3fbe50-f114-4e40-9e98-097a31d2251b/volumes"
Nov 02 12:44:29.049445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:29.049410    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:30.125392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:30.125354    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:30.273702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:30.273667    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:30.405690 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:30.405594    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:30.588358 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:30.588314    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:30.956295 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:30.956258    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:31.200659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:31.200618    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:31.423876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:31.423844    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:31.686277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:31.686163    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:33.030537 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:33.030504    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:34.340542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:34.340511    5879 scope.go:122] "RemoveContainer" containerID="5b284d5e01a038bdebc6a5f48998b5cbc1f94e3218c093560965b93a4c5da84d"
Nov 02 12:44:34.351108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:34.351089    5879 scope.go:122] "RemoveContainer" containerID="bbd6024cb0a8cfe632a4360cb9b34dfef51ce80b23758ecad17c398662b89acf"
Nov 02 12:44:34.360940 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:34.360914    5879 scope.go:122] "RemoveContainer" containerID="ca03b8de5f5057ed5e3af948cbb1553a780ac8b92ddbbc83ab5ab4423e73f970"
Nov 02 12:44:34.371098 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:34.371075    5879 scope.go:122] "RemoveContainer" containerID="534b8b6c75e5777bf0564f01761b8ecc87b88fd364018ae67db0ce11b558fc6a"
Nov 02 12:44:34.382078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:34.382053    5879 scope.go:122] "RemoveContainer" containerID="51d5f0ecbf1eec4f01aa93fc3abbf811663274dff4d4ff061b59fa54a461aeaa"
Nov 02 12:44:34.391918 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:34.391894    5879 scope.go:122] "RemoveContainer" containerID="3591cf5b44e3f73e597cefb7314d0374847032bb0dcb68384418504a3a5d23dc"
Nov 02 12:44:34.401593 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:34.401569    5879 scope.go:122] "RemoveContainer" containerID="72b5fc70259abbec7488ed64ddb5cf24154797bc61451e1aa726a27939d8a6fe"
Nov 02 12:44:34.644409 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:34.644374    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:34.773830 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:34.773797    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:34.910439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:34.910357    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:35.040347 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:35.040312    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:35.706853 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:35.706821    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:35.852616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:35.852582    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:36.717916 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:36.717883    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:36.872637 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:36.872602    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:37.126460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:37.126425    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:37.655570 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:37.655518    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:37.849505 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:37.849471    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:38.032407 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.032312    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:38.253519 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.253481    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:38.613462 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.613430    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:38.768919 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.768883    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:38.893835 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.893761    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:38.925876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.925845    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"]
Nov 02 12:44:38.926515 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.926500    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d11c3ef5-d6b3-45f9-a495-3119c068787f" containerName="revoke-keys"
Nov 02 12:44:38.926572 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.926516    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11c3ef5-d6b3-45f9-a495-3119c068787f" containerName="revoke-keys"
Nov 02 12:44:38.926650 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.926642    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11c3ef5-d6b3-45f9-a495-3119c068787f" containerName="revoke-keys"
Nov 02 12:44:38.932805 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.932783    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"
Nov 02 12:44:38.951074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.951041    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"]
Nov 02 12:44:38.959228 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.959158    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0cf7e\""
Nov 02 12:44:38.959403 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:38.959308    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0cf7e-dockercfg-wdm9l\""
Nov 02 12:44:39.004511 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.004466    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"]
Nov 02 12:44:39.005080 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.004983    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l" podUID="62a8dae1-0b29-4f71-9603-9793e5f50e28" containerName="istio-proxy" containerID="cri-o://7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef" gracePeriod=30
Nov 02 12:44:39.027236 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.027198    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zttlg\" (UniqueName: \"kubernetes.io/projected/277edcf7-7653-4f7f-b5af-958179c181c3-kube-api-access-zttlg\") pod \"payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg\" (UID: \"277edcf7-7653-4f7f-b5af-958179c181c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"
Nov 02 12:44:39.027407 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.027253    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277edcf7-7653-4f7f-b5af-958179c181c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg\" (UID: \"277edcf7-7653-4f7f-b5af-958179c181c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"
Nov 02 12:44:39.033827 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.033788    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"]
Nov 02 12:44:39.039299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.039273    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"
Nov 02 12:44:39.059687 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.058878    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"]
Nov 02 12:44:39.063978 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.063937    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:39.164066 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.158957    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92kl\" (UniqueName: \"kubernetes.io/projected/6ab41884-1345-483e-ab7d-362d118773bc-kube-api-access-c92kl\") pod \"payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj\" (UID: \"6ab41884-1345-483e-ab7d-362d118773bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"
Nov 02 12:44:39.164066 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.159059    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277edcf7-7653-4f7f-b5af-958179c181c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg\" (UID: \"277edcf7-7653-4f7f-b5af-958179c181c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"
Nov 02 12:44:39.164066 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.159144    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab41884-1345-483e-ab7d-362d118773bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj\" (UID: \"6ab41884-1345-483e-ab7d-362d118773bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"
Nov 02 12:44:39.164066 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.159176    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zttlg\" (UniqueName: \"kubernetes.io/projected/277edcf7-7653-4f7f-b5af-958179c181c3-kube-api-access-zttlg\") pod \"payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg\" (UID: \"277edcf7-7653-4f7f-b5af-958179c181c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"
Nov 02 12:44:39.168019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.164590    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277edcf7-7653-4f7f-b5af-958179c181c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg\" (UID: \"277edcf7-7653-4f7f-b5af-958179c181c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"
Nov 02 12:44:39.190023 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.189735    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zttlg\" (UniqueName: \"kubernetes.io/projected/277edcf7-7653-4f7f-b5af-958179c181c3-kube-api-access-zttlg\") pod \"payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg\" (UID: \"277edcf7-7653-4f7f-b5af-958179c181c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"
Nov 02 12:44:39.191060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.190484    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:39.254162 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.254131    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"
Nov 02 12:44:39.260692 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.260661    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab41884-1345-483e-ab7d-362d118773bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj\" (UID: \"6ab41884-1345-483e-ab7d-362d118773bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"
Nov 02 12:44:39.261033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.261013    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92kl\" (UniqueName: \"kubernetes.io/projected/6ab41884-1345-483e-ab7d-362d118773bc-kube-api-access-c92kl\") pod \"payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj\" (UID: \"6ab41884-1345-483e-ab7d-362d118773bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"
Nov 02 12:44:39.261562 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.261538    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab41884-1345-483e-ab7d-362d118773bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj\" (UID: \"6ab41884-1345-483e-ab7d-362d118773bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"
Nov 02 12:44:39.291254 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.291221    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92kl\" (UniqueName: \"kubernetes.io/projected/6ab41884-1345-483e-ab7d-362d118773bc-kube-api-access-c92kl\") pod \"payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj\" (UID: \"6ab41884-1345-483e-ab7d-362d118773bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"
Nov 02 12:44:39.368052 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.366332    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"
Nov 02 12:44:39.583150 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:44:39.583118    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277edcf7_7653_4f7f_b5af_958179c181c3.slice/crio-c9b53c9c6e4e7a01aff1d27a222a09981b291ae329e67f5bd07cf174f8b5c7b4 WatchSource:0}: Error finding container c9b53c9c6e4e7a01aff1d27a222a09981b291ae329e67f5bd07cf174f8b5c7b4: Status 404 returned error can't find the container with id c9b53c9c6e4e7a01aff1d27a222a09981b291ae329e67f5bd07cf174f8b5c7b4
Nov 02 12:44:39.584507 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.584474    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"]
Nov 02 12:44:39.697104 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:44:39.697033    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab41884_1345_483e_ab7d_362d118773bc.slice/crio-ee454da3c48768b9f2b2516f3c7db2532e82ea83004805777a5c8fedc19cf3a2 WatchSource:0}: Error finding container ee454da3c48768b9f2b2516f3c7db2532e82ea83004805777a5c8fedc19cf3a2: Status 404 returned error can't find the container with id ee454da3c48768b9f2b2516f3c7db2532e82ea83004805777a5c8fedc19cf3a2
Nov 02 12:44:39.698512 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.698485    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"]
Nov 02 12:44:39.726437 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.726411    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg" event={"ID":"277edcf7-7653-4f7f-b5af-958179c181c3","Type":"ContainerStarted","Data":"1adbed4ba6b077d865f007511c0e5fbda10434c34ed671bff6df4f07748f7f35"}
Nov 02 12:44:39.726574 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.726466    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg" event={"ID":"277edcf7-7653-4f7f-b5af-958179c181c3","Type":"ContainerStarted","Data":"c9b53c9c6e4e7a01aff1d27a222a09981b291ae329e67f5bd07cf174f8b5c7b4"}
Nov 02 12:44:39.726643 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.726610    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"
Nov 02 12:44:39.727629 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.727613    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj" event={"ID":"6ab41884-1345-483e-ab7d-362d118773bc","Type":"ContainerStarted","Data":"ee454da3c48768b9f2b2516f3c7db2532e82ea83004805777a5c8fedc19cf3a2"}
Nov 02 12:44:39.774362 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.774310    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg" podStartSLOduration=1.774295603 podStartE2EDuration="1.774295603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:44:39.772553489 +0000 UTC m=+1025.879256341" watchObservedRunningTime="2026-11-02 12:44:39.774295603 +0000 UTC m=+1025.880998459"
Nov 02 12:44:39.836488 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:39.836453    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:40.737297 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:40.737257    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj" event={"ID":"6ab41884-1345-483e-ab7d-362d118773bc","Type":"ContainerStarted","Data":"793397b3a7edb0a22e228c14755d9473ea5dfe8ec4417ac6f3747fae16df4dec"}
Nov 02 12:44:40.759957 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:40.759929    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:40.796168 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:40.796103    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj" podStartSLOduration=2.79608112 podStartE2EDuration="2.79608112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:44:40.780243902 +0000 UTC m=+1026.886946778" watchObservedRunningTime="2026-11-02 12:44:40.79608112 +0000 UTC m=+1026.902783973"
Nov 02 12:44:41.744637 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:41.744598    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"
Nov 02 12:44:42.518813 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:42.518768    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:42.664317 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:42.664279    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:43.808391 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:43.808325    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:43.990801 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:43.990750    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:44.286795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.286768    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:44:44.342328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.342297    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-socket\") pod \"62a8dae1-0b29-4f71-9603-9793e5f50e28\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") "
Nov 02 12:44:44.342513 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.342489    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-envoy\") pod \"62a8dae1-0b29-4f71-9603-9793e5f50e28\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") "
Nov 02 12:44:44.342616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.342601    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-data\") pod \"62a8dae1-0b29-4f71-9603-9793e5f50e28\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") "
Nov 02 12:44:44.342675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.342611    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-socket" pod "62a8dae1-0b29-4f71-9603-9793e5f50e28" (UID: "62a8dae1-0b29-4f71-9603-9793e5f50e28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.342675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.342654    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-certs\") pod \"62a8dae1-0b29-4f71-9603-9793e5f50e28\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") "
Nov 02 12:44:44.342782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.342686    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-kube-api-access-msvt4\" (UniqueName: \"kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-kube-api-access-msvt4\") pod \"62a8dae1-0b29-4f71-9603-9793e5f50e28\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") "
Nov 02 12:44:44.342782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.342724    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a8dae1-0b29-4f71-9603-9793e5f50e28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a8dae1-0b29-4f71-9603-9793e5f50e28-istiod-ca-cert\") pod \"62a8dae1-0b29-4f71-9603-9793e5f50e28\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") "
Nov 02 12:44:44.342782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.342758    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-token\" (UniqueName: \"kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-token\") pod \"62a8dae1-0b29-4f71-9603-9793e5f50e28\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") "
Nov 02 12:44:44.342934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.342835    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-podinfo\") pod \"62a8dae1-0b29-4f71-9603-9793e5f50e28\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") "
Nov 02 12:44:44.342934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.342878    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-data" pod "62a8dae1-0b29-4f71-9603-9793e5f50e28" (UID: "62a8dae1-0b29-4f71-9603-9793e5f50e28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.343071 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.342953    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-credential-socket\") pod \"62a8dae1-0b29-4f71-9603-9793e5f50e28\" (UID: \"62a8dae1-0b29-4f71-9603-9793e5f50e28\") "
Nov 02 12:44:44.343945 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.343916    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.344094 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.343949    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.344094 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.344037    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a8dae1-0b29-4f71-9603-9793e5f50e28-istiod-ca-cert" pod "62a8dae1-0b29-4f71-9603-9793e5f50e28" (UID: "62a8dae1-0b29-4f71-9603-9793e5f50e28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:44:44.344215 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.344105    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-credential-socket" pod "62a8dae1-0b29-4f71-9603-9793e5f50e28" (UID: "62a8dae1-0b29-4f71-9603-9793e5f50e28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.344430 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.344402    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-certs" pod "62a8dae1-0b29-4f71-9603-9793e5f50e28" (UID: "62a8dae1-0b29-4f71-9603-9793e5f50e28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.345875 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.345807    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-podinfo" pod "62a8dae1-0b29-4f71-9603-9793e5f50e28" (UID: "62a8dae1-0b29-4f71-9603-9793e5f50e28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:44:44.347148 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.347127    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-token" pod "62a8dae1-0b29-4f71-9603-9793e5f50e28" (UID: "62a8dae1-0b29-4f71-9603-9793e5f50e28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:44.347256 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.347160    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-kube-api-access-msvt4" pod "62a8dae1-0b29-4f71-9603-9793e5f50e28" (UID: "62a8dae1-0b29-4f71-9603-9793e5f50e28"). InnerVolumeSpecName "kube-api-access-msvt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:44:44.352559 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.352529    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-envoy" pod "62a8dae1-0b29-4f71-9603-9793e5f50e28" (UID: "62a8dae1-0b29-4f71-9603-9793e5f50e28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:44:44.410635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.410555    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:44.446426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.446362    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.446426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.446408    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.446426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.446426    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.446426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.446439    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a8dae1-0b29-4f71-9603-9793e5f50e28-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.446857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.446451    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msvt4\" (UniqueName: \"kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-kube-api-access-msvt4\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.446857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.446467    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a8dae1-0b29-4f71-9603-9793e5f50e28-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.446857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.446479    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a8dae1-0b29-4f71-9603-9793e5f50e28-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:44:44.681944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.681860    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:44.773377 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.773340    5879 generic.go:356] "Generic (PLEG): container finished" podID="62a8dae1-0b29-4f71-9603-9793e5f50e28" containerID="7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef" exitCode=0
Nov 02 12:44:44.773555 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.773393    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"
Nov 02 12:44:44.773555 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.773421    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l" event={"ID":"62a8dae1-0b29-4f71-9603-9793e5f50e28","Type":"ContainerDied","Data":"7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef"}
Nov 02 12:44:44.773555 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.773485    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l" event={"ID":"62a8dae1-0b29-4f71-9603-9793e5f50e28","Type":"ContainerDied","Data":"e1c93cc6fc082b2a6b4cdd0a0d3cf08aff170a7bc4d6137db9f5438016ba5d82"}
Nov 02 12:44:44.773555 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.773517    5879 scope.go:122] "RemoveContainer" containerID="7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef"
Nov 02 12:44:44.787720 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.787701    5879 scope.go:122] "RemoveContainer" containerID="7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef"
Nov 02 12:44:44.788139 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:44:44.788112    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef\": container with ID starting with 7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef not found: ID does not exist" containerID="7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef"
Nov 02 12:44:44.788216 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.788149    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef"} err="failed to get container status \"7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef\": rpc error: code = NotFound desc = could not find container \"7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef\": container with ID starting with 7c4605811706cb9ae612e0b612185bf04781acee67ac56c2e6789ea546efe8ef not found: ID does not exist"
Nov 02 12:44:44.828648 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:44:44.828614    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:44:44.832582 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.832545    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"]
Nov 02 12:44:44.840586 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:44.840551    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9f7f01-openshift-default-64d8f9568c-zd79l"]
Nov 02 12:44:45.050055 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:45.049938    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:45.831340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:45.831292    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:45.956481 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:45.956447    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:46.019592 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:46.019561    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a8dae1-0b29-4f71-9603-9793e5f50e28" path="/var/lib/kubelet/pods/62a8dae1-0b29-4f71-9603-9793e5f50e28/volumes"
Nov 02 12:44:46.457650 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:46.457422    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:46.918799 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:46.918767    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:48.032567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:48.032533    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"]
Nov 02 12:44:48.045100 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:48.045069    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b60761-8qj58"]
Nov 02 12:44:48.164727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:48.164686    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:49.820095 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:49.820065    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:50.030273 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:50.030240    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ecddb26-4e40-4c73-a1cb-01d157c8e8ce" path="/var/lib/kubelet/pods/0ecddb26-4e40-4c73-a1cb-01d157c8e8ce/volumes"
Nov 02 12:44:50.740432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:50.740406    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"
Nov 02 12:44:51.399984 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:51.399928    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:51.543272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:51.543237    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:52.732950 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:52.732906    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:52.755741 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:52.755697    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"
Nov 02 12:44:52.862335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:52.862295    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:53.412054 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:53.409552    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:54.049800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:54.049766    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:55.140058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:55.140009    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:56.642437 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:56.642399    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:58.051189 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:58.051153    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:58.190292 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:58.190259    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:58.644923 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:58.644874    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:44:58.775440 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:44:58.775402    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:00.079437 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.079360    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"]
Nov 02 12:45:00.079437 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.079401    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d0404ed-v4lm7"]
Nov 02 12:45:00.082370 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.082341    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:00.164978 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.164940    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"]
Nov 02 12:45:00.165575 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.165560    5879 cpu_manager.go:414] "containerMap: removing container" podUID="62a8dae1-0b29-4f71-9603-9793e5f50e28" containerName="istio-proxy"
Nov 02 12:45:00.165627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.165577    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a8dae1-0b29-4f71-9603-9793e5f50e28" containerName="istio-proxy"
Nov 02 12:45:00.165723 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.165714    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a8dae1-0b29-4f71-9603-9793e5f50e28" containerName="istio-proxy"
Nov 02 12:45:00.170078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.170054    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"
Nov 02 12:45:00.179373 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.178786    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"]
Nov 02 12:45:00.189512 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.189477    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"]
Nov 02 12:45:00.189803 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.189777    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"
Nov 02 12:45:00.195248 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.194314    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:45:00.206183 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.206158    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"]
Nov 02 12:45:00.226147 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.226113    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"
Nov 02 12:45:00.240539 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.240458    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"]
Nov 02 12:45:00.241404 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.241381    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"]
Nov 02 12:45:00.243141 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.242949    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"
Nov 02 12:45:00.245282 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.245255    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:00.250699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.250671    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"]
Nov 02 12:45:00.250838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.250721    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"]
Nov 02 12:45:00.250838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.250751    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"]
Nov 02 12:45:00.250961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.250843    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"
Nov 02 12:45:00.257276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.257245    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"]
Nov 02 12:45:00.297484 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.297443    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzrx\" (UniqueName: \"kubernetes.io/projected/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-kube-api-access-6jzrx\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s\" (UID: \"651a21a0-5eaf-4aa0-a3fc-d8760e824cba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"
Nov 02 12:45:00.297800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.297770    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjq6\" (UniqueName: \"kubernetes.io/projected/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-kube-api-access-msjq6\") pod \"maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw\" (UID: \"c7e3d2f9-00fe-4626-aa7c-c9639667deb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"
Nov 02 12:45:00.297918 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.297815    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6klhr\" (UniqueName: \"kubernetes.io/projected/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-kube-api-access-6klhr\") pod \"maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm\" (UID: \"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"
Nov 02 12:45:00.297967 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.297936    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6j47\" (UniqueName: \"kubernetes.io/projected/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-kube-api-access-c6j47\") pod \"maas-api-key-cleanup-29893725-96lz9\" (UID: \"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"
Nov 02 12:45:00.298038 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.297977    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj2k2\" (UniqueName: \"kubernetes.io/projected/cbb0f327-27f6-4bd4-99b5-85549548699e-kube-api-access-dj2k2\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh\" (UID: \"cbb0f327-27f6-4bd4-99b5-85549548699e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"
Nov 02 12:45:00.298086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.298057    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm\" (UID: \"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"
Nov 02 12:45:00.298210 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.298196    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s\" (UID: \"651a21a0-5eaf-4aa0-a3fc-d8760e824cba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"
Nov 02 12:45:00.298259 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.298247    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb0f327-27f6-4bd4-99b5-85549548699e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh\" (UID: \"cbb0f327-27f6-4bd4-99b5-85549548699e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"
Nov 02 12:45:00.298351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.298326    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893725-96lz9\" (UID: \"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"
Nov 02 12:45:00.298472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.298400    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw\" (UID: \"c7e3d2f9-00fe-4626-aa7c-c9639667deb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"
Nov 02 12:45:00.386282 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.386248    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:00.399702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.399666    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjq6\" (UniqueName: \"kubernetes.io/projected/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-kube-api-access-msjq6\") pod \"maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw\" (UID: \"c7e3d2f9-00fe-4626-aa7c-c9639667deb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"
Nov 02 12:45:00.399702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.399705    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6klhr\" (UniqueName: \"kubernetes.io/projected/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-kube-api-access-6klhr\") pod \"maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm\" (UID: \"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"
Nov 02 12:45:00.399948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.399770    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6j47\" (UniqueName: \"kubernetes.io/projected/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-kube-api-access-c6j47\") pod \"maas-api-key-cleanup-29893725-96lz9\" (UID: \"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"
Nov 02 12:45:00.399948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.399801    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj2k2\" (UniqueName: \"kubernetes.io/projected/cbb0f327-27f6-4bd4-99b5-85549548699e-kube-api-access-dj2k2\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh\" (UID: \"cbb0f327-27f6-4bd4-99b5-85549548699e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"
Nov 02 12:45:00.399948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.399827    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm\" (UID: \"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"
Nov 02 12:45:00.399948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.399914    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s\" (UID: \"651a21a0-5eaf-4aa0-a3fc-d8760e824cba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"
Nov 02 12:45:00.400226 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.399982    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb0f327-27f6-4bd4-99b5-85549548699e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh\" (UID: \"cbb0f327-27f6-4bd4-99b5-85549548699e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"
Nov 02 12:45:00.400226 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.400058    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893725-96lz9\" (UID: \"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"
Nov 02 12:45:00.400226 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.400109    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw\" (UID: \"c7e3d2f9-00fe-4626-aa7c-c9639667deb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"
Nov 02 12:45:00.400226 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.400219    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzrx\" (UniqueName: \"kubernetes.io/projected/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-kube-api-access-6jzrx\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s\" (UID: \"651a21a0-5eaf-4aa0-a3fc-d8760e824cba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"
Nov 02 12:45:00.400879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.400857    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm\" (UID: \"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"
Nov 02 12:45:00.400879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.400868    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s\" (UID: \"651a21a0-5eaf-4aa0-a3fc-d8760e824cba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"
Nov 02 12:45:00.401077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.400855    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893725-96lz9\" (UID: \"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"
Nov 02 12:45:00.401077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.400868    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb0f327-27f6-4bd4-99b5-85549548699e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh\" (UID: \"cbb0f327-27f6-4bd4-99b5-85549548699e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"
Nov 02 12:45:00.401077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.400920    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw\" (UID: \"c7e3d2f9-00fe-4626-aa7c-c9639667deb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"
Nov 02 12:45:00.426753 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.426721    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj2k2\" (UniqueName: \"kubernetes.io/projected/cbb0f327-27f6-4bd4-99b5-85549548699e-kube-api-access-dj2k2\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh\" (UID: \"cbb0f327-27f6-4bd4-99b5-85549548699e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"
Nov 02 12:45:00.426753 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.426740    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzrx\" (UniqueName: \"kubernetes.io/projected/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-kube-api-access-6jzrx\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s\" (UID: \"651a21a0-5eaf-4aa0-a3fc-d8760e824cba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"
Nov 02 12:45:00.426965 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.426809    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6j47\" (UniqueName: \"kubernetes.io/projected/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-kube-api-access-c6j47\") pod \"maas-api-key-cleanup-29893725-96lz9\" (UID: \"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"
Nov 02 12:45:00.431142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.431123    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjq6\" (UniqueName: \"kubernetes.io/projected/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-kube-api-access-msjq6\") pod \"maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw\" (UID: \"c7e3d2f9-00fe-4626-aa7c-c9639667deb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"
Nov 02 12:45:00.431423 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.431403    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6klhr\" (UniqueName: \"kubernetes.io/projected/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-kube-api-access-6klhr\") pod \"maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm\" (UID: \"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"
Nov 02 12:45:00.503037 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.502848    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"
Nov 02 12:45:00.522279 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.522253    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"
Nov 02 12:45:00.552522 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.552484    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"
Nov 02 12:45:00.569741 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.569702    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"
Nov 02 12:45:00.586332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.586068    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"
Nov 02 12:45:00.753196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.753164    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:00.825341 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:45:00.825311    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb0f327_27f6_4bd4_99b5_85549548699e.slice/crio-d845b5e51a9815e5207e6ad3bba924db33342d0074d2b48e65e859efde3177e4 WatchSource:0}: Error finding container d845b5e51a9815e5207e6ad3bba924db33342d0074d2b48e65e859efde3177e4: Status 404 returned error can't find the container with id d845b5e51a9815e5207e6ad3bba924db33342d0074d2b48e65e859efde3177e4
Nov 02 12:45:00.829092 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.829053    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"]
Nov 02 12:45:00.859893 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:45:00.859862    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651a21a0_5eaf_4aa0_a3fc_d8760e824cba.slice/crio-0a3dc98d58fe3d5b8b6b8b721f552407e1116eac500bb623bdce44ed4cd64311 WatchSource:0}: Error finding container 0a3dc98d58fe3d5b8b6b8b721f552407e1116eac500bb623bdce44ed4cd64311: Status 404 returned error can't find the container with id 0a3dc98d58fe3d5b8b6b8b721f552407e1116eac500bb623bdce44ed4cd64311
Nov 02 12:45:00.865768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.864550    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"]
Nov 02 12:45:00.920893 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.920858    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh" event={"ID":"cbb0f327-27f6-4bd4-99b5-85549548699e","Type":"ContainerStarted","Data":"d845b5e51a9815e5207e6ad3bba924db33342d0074d2b48e65e859efde3177e4"}
Nov 02 12:45:00.923267 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.923208    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s" event={"ID":"651a21a0-5eaf-4aa0-a3fc-d8760e824cba","Type":"ContainerStarted","Data":"0a3dc98d58fe3d5b8b6b8b721f552407e1116eac500bb623bdce44ed4cd64311"}
Nov 02 12:45:00.925700 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.925651    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"]
Nov 02 12:45:00.932874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.932849    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"]
Nov 02 12:45:00.937253 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:00.937230    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"]
Nov 02 12:45:00.992231 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:45:00.992142    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b650e3c_7ad6_4eb9_8f0d_14206e9046dc.slice/crio-c4a81f0893b5c0338ab4e06e18213b116d5f13d7ae370c0c38fb2a3f21683ef8 WatchSource:0}: Error finding container c4a81f0893b5c0338ab4e06e18213b116d5f13d7ae370c0c38fb2a3f21683ef8: Status 404 returned error can't find the container with id c4a81f0893b5c0338ab4e06e18213b116d5f13d7ae370c0c38fb2a3f21683ef8
Nov 02 12:45:00.992665 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:45:00.992456    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e3d2f9_00fe_4626_aa7c_c9639667deb2.slice/crio-a670b60e91b73b39e97996e8e98e89d885568ff28016a7ac479fa82b103e9a3e WatchSource:0}: Error finding container a670b60e91b73b39e97996e8e98e89d885568ff28016a7ac479fa82b103e9a3e: Status 404 returned error can't find the container with id a670b60e91b73b39e97996e8e98e89d885568ff28016a7ac479fa82b103e9a3e
Nov 02 12:45:00.992818 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:45:00.992793    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ffa5b3_83e8_4acc_9e31_8bc377c3aa09.slice/crio-cec30e7e14f808c9c4d1b8976fecd3571d14d2b0133b6d89f16e4d651c34e91c WatchSource:0}: Error finding container cec30e7e14f808c9c4d1b8976fecd3571d14d2b0133b6d89f16e4d651c34e91c: Status 404 returned error can't find the container with id cec30e7e14f808c9c4d1b8976fecd3571d14d2b0133b6d89f16e4d651c34e91c
Nov 02 12:45:01.130653 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:01.130581    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:01.945650 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:01.945608    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw" event={"ID":"c7e3d2f9-00fe-4626-aa7c-c9639667deb2","Type":"ContainerStarted","Data":"565200d5eeca84a871935e290a94eef7db0b59c381fb3cf6c8c8e18a3b745cf1"}
Nov 02 12:45:01.945831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:01.945750    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw" event={"ID":"c7e3d2f9-00fe-4626-aa7c-c9639667deb2","Type":"ContainerStarted","Data":"a670b60e91b73b39e97996e8e98e89d885568ff28016a7ac479fa82b103e9a3e"}
Nov 02 12:45:01.947266 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:01.947237    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9" event={"ID":"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09","Type":"ContainerStarted","Data":"874442c5c99972f0f693dd1e676b74fe7af501d93a25efbc26e5f18ae4d9b7be"}
Nov 02 12:45:01.947404 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:01.947291    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9" event={"ID":"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09","Type":"ContainerStarted","Data":"cec30e7e14f808c9c4d1b8976fecd3571d14d2b0133b6d89f16e4d651c34e91c"}
Nov 02 12:45:01.948752 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:01.948728    5879 generic.go:356] "Generic (PLEG): container finished" podID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" containerID="fd64e2606786dd8f021a78cf2672880f13335f311435cb17b917355e843e47dd" exitCode=60
Nov 02 12:45:01.948837 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:01.948807    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s" event={"ID":"651a21a0-5eaf-4aa0-a3fc-d8760e824cba","Type":"ContainerDied","Data":"fd64e2606786dd8f021a78cf2672880f13335f311435cb17b917355e843e47dd"}
Nov 02 12:45:01.948977 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:01.948953    5879 scope.go:122] "RemoveContainer" containerID="fd64e2606786dd8f021a78cf2672880f13335f311435cb17b917355e843e47dd"
Nov 02 12:45:01.950810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:01.950780    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" event={"ID":"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc","Type":"ContainerStarted","Data":"8887d458ff266a478794d4c0a1de51f914230e0dc1dde6c584347adc3fd4725f"}
Nov 02 12:45:01.950902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:01.950827    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" event={"ID":"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc","Type":"ContainerStarted","Data":"c4a81f0893b5c0338ab4e06e18213b116d5f13d7ae370c0c38fb2a3f21683ef8"}
Nov 02 12:45:01.952433 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:01.952327    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh" event={"ID":"cbb0f327-27f6-4bd4-99b5-85549548699e","Type":"ContainerStarted","Data":"a80908c9cfe63245225d2d6ff98359622ead72099d5303ba79bfc113f84fa412"}
Nov 02 12:45:02.005223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.005172    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw" podStartSLOduration=2.00515235 podStartE2EDuration="2.00515235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:02.002652446 +0000 UTC m=+1048.109355299" watchObservedRunningTime="2026-11-02 12:45:02.00515235 +0000 UTC m=+1048.111855201"
Nov 02 12:45:02.019407 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.019370    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46e8118-e52c-4a9b-80ab-fae8692e228d" path="/var/lib/kubelet/pods/d46e8118-e52c-4a9b-80ab-fae8692e228d/volumes"
Nov 02 12:45:02.078112 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.078040    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh" podStartSLOduration=2.078025691 podStartE2EDuration="2.078025691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:02.075050726 +0000 UTC m=+1048.181753577" watchObservedRunningTime="2026-11-02 12:45:02.078025691 +0000 UTC m=+1048.184728536"
Nov 02 12:45:02.112467 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.112430    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:02.166665 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.166556    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9" podStartSLOduration=2.16654186 podStartE2EDuration="2.16654186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:02.165035272 +0000 UTC m=+1048.271738119" watchObservedRunningTime="2026-11-02 12:45:02.16654186 +0000 UTC m=+1048.273244711"
Nov 02 12:45:02.213899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.213822    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" podStartSLOduration=2.213806988 podStartE2EDuration="2.213806988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:02.209987969 +0000 UTC m=+1048.316690825" watchObservedRunningTime="2026-11-02 12:45:02.213806988 +0000 UTC m=+1048.320509885"
Nov 02 12:45:02.245382 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.245355    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:02.700841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.700702    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:02.908873 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.908838    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:02.965370 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.965275    5879 generic.go:356] "Generic (PLEG): container finished" podID="c7e3d2f9-00fe-4626-aa7c-c9639667deb2" containerID="565200d5eeca84a871935e290a94eef7db0b59c381fb3cf6c8c8e18a3b745cf1" exitCode=60
Nov 02 12:45:02.965538 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.965365    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw" event={"ID":"c7e3d2f9-00fe-4626-aa7c-c9639667deb2","Type":"ContainerDied","Data":"565200d5eeca84a871935e290a94eef7db0b59c381fb3cf6c8c8e18a3b745cf1"}
Nov 02 12:45:02.965779 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.965728    5879 scope.go:122] "RemoveContainer" containerID="565200d5eeca84a871935e290a94eef7db0b59c381fb3cf6c8c8e18a3b745cf1"
Nov 02 12:45:02.968149 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.968126    5879 generic.go:356] "Generic (PLEG): container finished" podID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" containerID="874442c5c99972f0f693dd1e676b74fe7af501d93a25efbc26e5f18ae4d9b7be" exitCode=60
Nov 02 12:45:02.968267 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.968248    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9" event={"ID":"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09","Type":"ContainerDied","Data":"874442c5c99972f0f693dd1e676b74fe7af501d93a25efbc26e5f18ae4d9b7be"}
Nov 02 12:45:02.968612 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.968596    5879 scope.go:122] "RemoveContainer" containerID="874442c5c99972f0f693dd1e676b74fe7af501d93a25efbc26e5f18ae4d9b7be"
Nov 02 12:45:02.973456 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.973388    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s" event={"ID":"651a21a0-5eaf-4aa0-a3fc-d8760e824cba","Type":"ContainerStarted","Data":"c65b09c54807914a8ccd54237426cd0b6214834e8dcd8345498ba2e78f91f715"}
Nov 02 12:45:02.975792 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.975771    5879 generic.go:356] "Generic (PLEG): container finished" podID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" containerID="8887d458ff266a478794d4c0a1de51f914230e0dc1dde6c584347adc3fd4725f" exitCode=60
Nov 02 12:45:02.975878 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.975846    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" event={"ID":"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc","Type":"ContainerDied","Data":"8887d458ff266a478794d4c0a1de51f914230e0dc1dde6c584347adc3fd4725f"}
Nov 02 12:45:02.976139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.976118    5879 scope.go:122] "RemoveContainer" containerID="8887d458ff266a478794d4c0a1de51f914230e0dc1dde6c584347adc3fd4725f"
Nov 02 12:45:02.977709 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.977688    5879 generic.go:356] "Generic (PLEG): container finished" podID="cbb0f327-27f6-4bd4-99b5-85549548699e" containerID="a80908c9cfe63245225d2d6ff98359622ead72099d5303ba79bfc113f84fa412" exitCode=60
Nov 02 12:45:02.977782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.977766    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh" event={"ID":"cbb0f327-27f6-4bd4-99b5-85549548699e","Type":"ContainerDied","Data":"a80908c9cfe63245225d2d6ff98359622ead72099d5303ba79bfc113f84fa412"}
Nov 02 12:45:02.977978 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:02.977966    5879 scope.go:122] "RemoveContainer" containerID="a80908c9cfe63245225d2d6ff98359622ead72099d5303ba79bfc113f84fa412"
Nov 02 12:45:03.059509 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:03.059445    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s" podStartSLOduration=3.059421321 podStartE2EDuration="3.059421321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:03.058475203 +0000 UTC m=+1049.165178054" watchObservedRunningTime="2026-11-02 12:45:03.059421321 +0000 UTC m=+1049.166124173"
Nov 02 12:45:03.946187 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:03.946128    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:03.991921 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:03.991889    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" event={"ID":"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc","Type":"ContainerStarted","Data":"de43a4050805b37d8a5ead4303793d50bbd629de32d26ba488a362ad608d6e7a"}
Nov 02 12:45:03.994057 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:03.994029    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh" event={"ID":"cbb0f327-27f6-4bd4-99b5-85549548699e","Type":"ContainerStarted","Data":"e19f1a7cb11a475028e16f16ad20bc82190be6eb927bfba943cc1f1428fa566d"}
Nov 02 12:45:03.995932 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:03.995911    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw" event={"ID":"c7e3d2f9-00fe-4626-aa7c-c9639667deb2","Type":"ContainerStarted","Data":"718ed9b4b584730dcad7029c6769b57ff7d26f8c4932bff595f3a6499cb91122"}
Nov 02 12:45:03.997797 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:03.997767    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9" event={"ID":"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09","Type":"ContainerStarted","Data":"a961e4aad529c03375169461a041cca4f6871dd63cb8ebe019542b3a79ac968e"}
Nov 02 12:45:04.000521 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:04.000495    5879 generic.go:356] "Generic (PLEG): container finished" podID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" containerID="c65b09c54807914a8ccd54237426cd0b6214834e8dcd8345498ba2e78f91f715" exitCode=60
Nov 02 12:45:04.000651 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:04.000595    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s" event={"ID":"651a21a0-5eaf-4aa0-a3fc-d8760e824cba","Type":"ContainerDied","Data":"c65b09c54807914a8ccd54237426cd0b6214834e8dcd8345498ba2e78f91f715"}
Nov 02 12:45:04.000651 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:04.000638    5879 scope.go:122] "RemoveContainer" containerID="fd64e2606786dd8f021a78cf2672880f13335f311435cb17b917355e843e47dd"
Nov 02 12:45:04.001432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:04.001132    5879 scope.go:122] "RemoveContainer" containerID="c65b09c54807914a8ccd54237426cd0b6214834e8dcd8345498ba2e78f91f715"
Nov 02 12:45:04.001432 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:04.001418    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s_odh-ai-gateway-infra(651a21a0-5eaf-4aa0-a3fc-d8760e824cba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba"
Nov 02 12:45:05.011491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.011458    5879 generic.go:356] "Generic (PLEG): container finished" podID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" containerID="de43a4050805b37d8a5ead4303793d50bbd629de32d26ba488a362ad608d6e7a" exitCode=60
Nov 02 12:45:05.011973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.011594    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" event={"ID":"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc","Type":"ContainerDied","Data":"de43a4050805b37d8a5ead4303793d50bbd629de32d26ba488a362ad608d6e7a"}
Nov 02 12:45:05.011973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.011645    5879 scope.go:122] "RemoveContainer" containerID="8887d458ff266a478794d4c0a1de51f914230e0dc1dde6c584347adc3fd4725f"
Nov 02 12:45:05.011973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.011784    5879 scope.go:122] "RemoveContainer" containerID="de43a4050805b37d8a5ead4303793d50bbd629de32d26ba488a362ad608d6e7a"
Nov 02 12:45:05.012203 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:05.012150    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm_odh-ai-gateway-infra(4b650e3c-7ad6-4eb9-8f0d-14206e9046dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc"
Nov 02 12:45:05.014116 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.014090    5879 generic.go:356] "Generic (PLEG): container finished" podID="cbb0f327-27f6-4bd4-99b5-85549548699e" containerID="e19f1a7cb11a475028e16f16ad20bc82190be6eb927bfba943cc1f1428fa566d" exitCode=60
Nov 02 12:45:05.014240 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.014181    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh" event={"ID":"cbb0f327-27f6-4bd4-99b5-85549548699e","Type":"ContainerDied","Data":"e19f1a7cb11a475028e16f16ad20bc82190be6eb927bfba943cc1f1428fa566d"}
Nov 02 12:45:05.014412 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.014394    5879 scope.go:122] "RemoveContainer" containerID="e19f1a7cb11a475028e16f16ad20bc82190be6eb927bfba943cc1f1428fa566d"
Nov 02 12:45:05.014732 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:05.014712    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh_odh-ai-gateway-infra(cbb0f327-27f6-4bd4-99b5-85549548699e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e"
Nov 02 12:45:05.016618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.016593    5879 generic.go:356] "Generic (PLEG): container finished" podID="c7e3d2f9-00fe-4626-aa7c-c9639667deb2" containerID="718ed9b4b584730dcad7029c6769b57ff7d26f8c4932bff595f3a6499cb91122" exitCode=60
Nov 02 12:45:05.016724 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.016688    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw" event={"ID":"c7e3d2f9-00fe-4626-aa7c-c9639667deb2","Type":"ContainerDied","Data":"718ed9b4b584730dcad7029c6769b57ff7d26f8c4932bff595f3a6499cb91122"}
Nov 02 12:45:05.016903 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.016889    5879 scope.go:122] "RemoveContainer" containerID="718ed9b4b584730dcad7029c6769b57ff7d26f8c4932bff595f3a6499cb91122"
Nov 02 12:45:05.017235 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:05.017213    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw_odh-ai-gateway-infra(c7e3d2f9-00fe-4626-aa7c-c9639667deb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw" podUID="c7e3d2f9-00fe-4626-aa7c-c9639667deb2"
Nov 02 12:45:05.019088 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.019066    5879 generic.go:356] "Generic (PLEG): container finished" podID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" containerID="a961e4aad529c03375169461a041cca4f6871dd63cb8ebe019542b3a79ac968e" exitCode=60
Nov 02 12:45:05.019181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.019150    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9" event={"ID":"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09","Type":"ContainerDied","Data":"a961e4aad529c03375169461a041cca4f6871dd63cb8ebe019542b3a79ac968e"}
Nov 02 12:45:05.019388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.019370    5879 scope.go:122] "RemoveContainer" containerID="a961e4aad529c03375169461a041cca4f6871dd63cb8ebe019542b3a79ac968e"
Nov 02 12:45:05.019712 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:05.019688    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893725-96lz9_odh-ai-gateway-infra(e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09"
Nov 02 12:45:05.021813 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.021797    5879 scope.go:122] "RemoveContainer" containerID="c65b09c54807914a8ccd54237426cd0b6214834e8dcd8345498ba2e78f91f715"
Nov 02 12:45:05.022088 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:05.022067    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s_odh-ai-gateway-infra(651a21a0-5eaf-4aa0-a3fc-d8760e824cba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba"
Nov 02 12:45:05.026847 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.026830    5879 scope.go:122] "RemoveContainer" containerID="a80908c9cfe63245225d2d6ff98359622ead72099d5303ba79bfc113f84fa412"
Nov 02 12:45:05.041330 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.041313    5879 scope.go:122] "RemoveContainer" containerID="565200d5eeca84a871935e290a94eef7db0b59c381fb3cf6c8c8e18a3b745cf1"
Nov 02 12:45:05.055667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.055644    5879 scope.go:122] "RemoveContainer" containerID="874442c5c99972f0f693dd1e676b74fe7af501d93a25efbc26e5f18ae4d9b7be"
Nov 02 12:45:05.436083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.436039    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:05.505451 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:05.505418    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"]
Nov 02 12:45:06.067084 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:06.057811    5879 scope.go:122] "RemoveContainer" containerID="a961e4aad529c03375169461a041cca4f6871dd63cb8ebe019542b3a79ac968e"
Nov 02 12:45:06.067084 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:06.058023    5879 scope.go:122] "RemoveContainer" containerID="de43a4050805b37d8a5ead4303793d50bbd629de32d26ba488a362ad608d6e7a"
Nov 02 12:45:06.067084 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:06.058383    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm_odh-ai-gateway-infra(4b650e3c-7ad6-4eb9-8f0d-14206e9046dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc"
Nov 02 12:45:06.067084 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:06.058432    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893725-96lz9_odh-ai-gateway-infra(e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09"
Nov 02 12:45:06.067084 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:06.059981    5879 scope.go:122] "RemoveContainer" containerID="e19f1a7cb11a475028e16f16ad20bc82190be6eb927bfba943cc1f1428fa566d"
Nov 02 12:45:06.067084 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:06.060337    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh_odh-ai-gateway-infra(cbb0f327-27f6-4bd4-99b5-85549548699e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e"
Nov 02 12:45:06.067084 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:06.063261    5879 scope.go:122] "RemoveContainer" containerID="718ed9b4b584730dcad7029c6769b57ff7d26f8c4932bff595f3a6499cb91122"
Nov 02 12:45:06.067084 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:06.063530    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw_odh-ai-gateway-infra(c7e3d2f9-00fe-4626-aa7c-c9639667deb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw" podUID="c7e3d2f9-00fe-4626-aa7c-c9639667deb2"
Nov 02 12:45:06.311071 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:06.310984    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:06.827233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:06.827164    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:06.951593 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:06.951548    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:07.278983 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:07.278956    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"
Nov 02 12:45:07.288041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:07.287983    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:07.437792 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:07.437700    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-maas-api-service-ca\") pod \"c7e3d2f9-00fe-4626-aa7c-c9639667deb2\" (UID: \"c7e3d2f9-00fe-4626-aa7c-c9639667deb2\") "
Nov 02 12:45:07.437792 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:07.437755    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-kube-api-access-msjq6\" (UniqueName: \"kubernetes.io/projected/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-kube-api-access-msjq6\") pod \"c7e3d2f9-00fe-4626-aa7c-c9639667deb2\" (UID: \"c7e3d2f9-00fe-4626-aa7c-c9639667deb2\") "
Nov 02 12:45:07.438165 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:07.438139    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-maas-api-service-ca" pod "c7e3d2f9-00fe-4626-aa7c-c9639667deb2" (UID: "c7e3d2f9-00fe-4626-aa7c-c9639667deb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:07.438492 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:07.438435    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:07.440339 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:07.440313    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-kube-api-access-msjq6" pod "c7e3d2f9-00fe-4626-aa7c-c9639667deb2" (UID: "c7e3d2f9-00fe-4626-aa7c-c9639667deb2"). InnerVolumeSpecName "kube-api-access-msjq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:07.540458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:07.540417    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msjq6\" (UniqueName: \"kubernetes.io/projected/c7e3d2f9-00fe-4626-aa7c-c9639667deb2-kube-api-access-msjq6\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:08.075857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:08.075826    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:08.089097 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:08.089070    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"
Nov 02 12:45:08.089097 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:08.089084    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw" event={"ID":"c7e3d2f9-00fe-4626-aa7c-c9639667deb2","Type":"ContainerDied","Data":"a670b60e91b73b39e97996e8e98e89d885568ff28016a7ac479fa82b103e9a3e"}
Nov 02 12:45:08.089371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:08.089153    5879 scope.go:122] "RemoveContainer" containerID="718ed9b4b584730dcad7029c6769b57ff7d26f8c4932bff595f3a6499cb91122"
Nov 02 12:45:08.146736 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:08.146617    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"]
Nov 02 12:45:08.161372 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:08.161341    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-6a3af1-29893725-n2rfw"]
Nov 02 12:45:09.973024 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:09.968154    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:10.018712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:10.018683    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7e3d2f9-00fe-4626-aa7c-c9639667deb2" path="/var/lib/kubelet/pods/c7e3d2f9-00fe-4626-aa7c-c9639667deb2/volumes"
Nov 02 12:45:10.103379 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:10.103345    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:10.240102 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:10.239969    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:10.470327 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:10.470290    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:10.819191 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:10.819157    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"]
Nov 02 12:45:10.874472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:10.874440    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"]
Nov 02 12:45:11.021522 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:11.021490    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:11.031728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:11.031699    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"]
Nov 02 12:45:11.036463 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:11.036437    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3173f45-bxrf2"]
Nov 02 12:45:11.150643 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:11.150609    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:11.334605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:11.334569    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:11.732979 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:11.732939    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:12.023810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:12.023719    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea33eed1-d3b6-40f9-8d5f-6542bd424e6a" path="/var/lib/kubelet/pods/ea33eed1-d3b6-40f9-8d5f-6542bd424e6a/volumes"
Nov 02 12:45:12.034407 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:12.034360    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:12.307826 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:12.307747    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:12.785956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:12.785908    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:13.338452 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:13.338418    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:14.821021 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:14.820954    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:45:15.014137 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:15.014105    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:15.820135 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:15.820093    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj" podUID="6ab41884-1345-483e-ab7d-362d118773bc" containerName="payload-processing" containerID="cri-o://793397b3a7edb0a22e228c14755d9473ea5dfe8ec4417ac6f3747fae16df4dec" gracePeriod=25
Nov 02 12:45:15.874754 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:15.874710    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg" podUID="277edcf7-7653-4f7f-b5af-958179c181c3" containerName="payload-pre-processing" containerID="cri-o://1adbed4ba6b077d865f007511c0e5fbda10434c34ed671bff6df4f07748f7f35" gracePeriod=25
Nov 02 12:45:16.016306 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.010192    5879 scope.go:122] "RemoveContainer" containerID="c65b09c54807914a8ccd54237426cd0b6214834e8dcd8345498ba2e78f91f715"
Nov 02 12:45:16.016306 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.011329    5879 scope.go:122] "RemoveContainer" containerID="a961e4aad529c03375169461a041cca4f6871dd63cb8ebe019542b3a79ac968e"
Nov 02 12:45:16.018390 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.016498    5879 scope.go:122] "RemoveContainer" containerID="e19f1a7cb11a475028e16f16ad20bc82190be6eb927bfba943cc1f1428fa566d"
Nov 02 12:45:16.097933 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.097903    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"]
Nov 02 12:45:16.099101 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.099077    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c7e3d2f9-00fe-4626-aa7c-c9639667deb2" containerName="cleanup"
Nov 02 12:45:16.099264 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.099247    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e3d2f9-00fe-4626-aa7c-c9639667deb2" containerName="cleanup"
Nov 02 12:45:16.099420 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.099406    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c7e3d2f9-00fe-4626-aa7c-c9639667deb2" containerName="cleanup"
Nov 02 12:45:16.099528 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.099514    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e3d2f9-00fe-4626-aa7c-c9639667deb2" containerName="cleanup"
Nov 02 12:45:16.099850 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.099832    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e3d2f9-00fe-4626-aa7c-c9639667deb2" containerName="cleanup"
Nov 02 12:45:16.099962 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.099952    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e3d2f9-00fe-4626-aa7c-c9639667deb2" containerName="cleanup"
Nov 02 12:45:16.105773 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.105753    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.147389 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.147364    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"]
Nov 02 12:45:16.169130 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.168679    5879 generic.go:356] "Generic (PLEG): container finished" podID="277edcf7-7653-4f7f-b5af-958179c181c3" containerID="1adbed4ba6b077d865f007511c0e5fbda10434c34ed671bff6df4f07748f7f35" exitCode=0
Nov 02 12:45:16.169130 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.168815    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg" event={"ID":"277edcf7-7653-4f7f-b5af-958179c181c3","Type":"ContainerDied","Data":"1adbed4ba6b077d865f007511c0e5fbda10434c34ed671bff6df4f07748f7f35"}
Nov 02 12:45:16.173794 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.173762    5879 generic.go:356] "Generic (PLEG): container finished" podID="6ab41884-1345-483e-ab7d-362d118773bc" containerID="793397b3a7edb0a22e228c14755d9473ea5dfe8ec4417ac6f3747fae16df4dec" exitCode=0
Nov 02 12:45:16.173900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.173869    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj" event={"ID":"6ab41884-1345-483e-ab7d-362d118773bc","Type":"ContainerDied","Data":"793397b3a7edb0a22e228c14755d9473ea5dfe8ec4417ac6f3747fae16df4dec"}
Nov 02 12:45:16.173973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.173918    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj" event={"ID":"6ab41884-1345-483e-ab7d-362d118773bc","Type":"ContainerDied","Data":"ee454da3c48768b9f2b2516f3c7db2532e82ea83004805777a5c8fedc19cf3a2"}
Nov 02 12:45:16.173973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.173944    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee454da3c48768b9f2b2516f3c7db2532e82ea83004805777a5c8fedc19cf3a2"
Nov 02 12:45:16.186365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.186338    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f0cf7e-openshift-default-dockercfg-s488n\""
Nov 02 12:45:16.201667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.201638    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"
Nov 02 12:45:16.249555 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.249523    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.249555 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.249557    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.249742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.249636    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.249785 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.249756    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.249819 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.249805    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e442f069-a92e-4ab0-b936-0060f44b9e80-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.249873 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.249862    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.249909 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.249901    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bntxl\" (UniqueName: \"kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-kube-api-access-bntxl\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.249982 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.249971    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.250060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.250042    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.308187 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.308158    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"
Nov 02 12:45:16.350626 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.350557    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab41884-1345-483e-ab7d-362d118773bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab41884-1345-483e-ab7d-362d118773bc-plugins-config-volume\") pod \"6ab41884-1345-483e-ab7d-362d118773bc\" (UID: \"6ab41884-1345-483e-ab7d-362d118773bc\") "
Nov 02 12:45:16.350771 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.350753    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab41884-1345-483e-ab7d-362d118773bc-kube-api-access-c92kl\" (UniqueName: \"kubernetes.io/projected/6ab41884-1345-483e-ab7d-362d118773bc-kube-api-access-c92kl\") pod \"6ab41884-1345-483e-ab7d-362d118773bc\" (UID: \"6ab41884-1345-483e-ab7d-362d118773bc\") "
Nov 02 12:45:16.350950 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.350925    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab41884-1345-483e-ab7d-362d118773bc-plugins-config-volume" pod "6ab41884-1345-483e-ab7d-362d118773bc" (UID: "6ab41884-1345-483e-ab7d-362d118773bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:16.351255 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.351236    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.351355 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.351315    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e442f069-a92e-4ab0-b936-0060f44b9e80-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.351355 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.351347    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.351467 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.351406    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bntxl\" (UniqueName: \"kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-kube-api-access-bntxl\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.351523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.351510    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.351571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.351537    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.351683 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.351668    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.351741 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.351698    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.351795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.351761    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.351795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.351769    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.351913 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.351893    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab41884-1345-483e-ab7d-362d118773bc-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:16.352037 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.352022    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.352106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.352091    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.352160 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.352099    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e442f069-a92e-4ab0-b936-0060f44b9e80-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.352220 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.352172    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.353135 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.353113    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab41884-1345-483e-ab7d-362d118773bc-kube-api-access-c92kl" pod "6ab41884-1345-483e-ab7d-362d118773bc" (UID: "6ab41884-1345-483e-ab7d-362d118773bc"). InnerVolumeSpecName "kube-api-access-c92kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:16.354092 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.354072    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.354182 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.354165    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.383522 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.383401    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bntxl\" (UniqueName: \"kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-kube-api-access-bntxl\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.386405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.386384    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.452634 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.452601    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277edcf7-7653-4f7f-b5af-958179c181c3-kube-api-access-zttlg\" (UniqueName: \"kubernetes.io/projected/277edcf7-7653-4f7f-b5af-958179c181c3-kube-api-access-zttlg\") pod \"277edcf7-7653-4f7f-b5af-958179c181c3\" (UID: \"277edcf7-7653-4f7f-b5af-958179c181c3\") "
Nov 02 12:45:16.452771 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.452743    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277edcf7-7653-4f7f-b5af-958179c181c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277edcf7-7653-4f7f-b5af-958179c181c3-plugins-config-volume\") pod \"277edcf7-7653-4f7f-b5af-958179c181c3\" (UID: \"277edcf7-7653-4f7f-b5af-958179c181c3\") "
Nov 02 12:45:16.453124 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.453096    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277edcf7-7653-4f7f-b5af-958179c181c3-plugins-config-volume" pod "277edcf7-7653-4f7f-b5af-958179c181c3" (UID: "277edcf7-7653-4f7f-b5af-958179c181c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:16.453343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.453330    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c92kl\" (UniqueName: \"kubernetes.io/projected/6ab41884-1345-483e-ab7d-362d118773bc-kube-api-access-c92kl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:16.453394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.453346    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/277edcf7-7653-4f7f-b5af-958179c181c3-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:16.454874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.454852    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277edcf7-7653-4f7f-b5af-958179c181c3-kube-api-access-zttlg" pod "277edcf7-7653-4f7f-b5af-958179c181c3" (UID: "277edcf7-7653-4f7f-b5af-958179c181c3"). InnerVolumeSpecName "kube-api-access-zttlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:16.488093 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.488045    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:16.526407 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.526374    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:16.558335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.558288    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zttlg\" (UniqueName: \"kubernetes.io/projected/277edcf7-7653-4f7f-b5af-958179c181c3-kube-api-access-zttlg\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:16.676647 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.676600    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:16.820888 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:45:16.820854    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode442f069_a92e_4ab0_b936_0060f44b9e80.slice/crio-9047a8b75beab120bf28562d860754af6f5a9b5e37bacc0115fcffc355638a8c WatchSource:0}: Error finding container 9047a8b75beab120bf28562d860754af6f5a9b5e37bacc0115fcffc355638a8c: Status 404 returned error can't find the container with id 9047a8b75beab120bf28562d860754af6f5a9b5e37bacc0115fcffc355638a8c
Nov 02 12:45:16.822987 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.822967    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"]
Nov 02 12:45:16.823967 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.823933    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:45:16.824092 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.824036    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:45:16.824092 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.824065    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:45:16.998573 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:16.998493    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:17.185204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.185174    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"
Nov 02 12:45:17.185391 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.185255    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg" event={"ID":"277edcf7-7653-4f7f-b5af-958179c181c3","Type":"ContainerDied","Data":"c9b53c9c6e4e7a01aff1d27a222a09981b291ae329e67f5bd07cf174f8b5c7b4"}
Nov 02 12:45:17.185391 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.185334    5879 scope.go:122] "RemoveContainer" containerID="1adbed4ba6b077d865f007511c0e5fbda10434c34ed671bff6df4f07748f7f35"
Nov 02 12:45:17.187656 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.187630    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9" event={"ID":"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09","Type":"ContainerStarted","Data":"0b2c761404a31f7c9a0f3c0bb38f9817ac1631fab617cc435e558364782a4489"}
Nov 02 12:45:17.193627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.193599    5879 generic.go:356] "Generic (PLEG): container finished" podID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" containerID="b17d6ea73000ba58890989314b02a027f6a2590a2d1d1ddc4d7f11930415997a" exitCode=60
Nov 02 12:45:17.193731 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.193665    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s" event={"ID":"651a21a0-5eaf-4aa0-a3fc-d8760e824cba","Type":"ContainerDied","Data":"b17d6ea73000ba58890989314b02a027f6a2590a2d1d1ddc4d7f11930415997a"}
Nov 02 12:45:17.194133 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.194115    5879 scope.go:122] "RemoveContainer" containerID="b17d6ea73000ba58890989314b02a027f6a2590a2d1d1ddc4d7f11930415997a"
Nov 02 12:45:17.194432 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:17.194411    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s_odh-ai-gateway-infra(651a21a0-5eaf-4aa0-a3fc-d8760e824cba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba"
Nov 02 12:45:17.195874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.195852    5879 generic.go:356] "Generic (PLEG): container finished" podID="cbb0f327-27f6-4bd4-99b5-85549548699e" containerID="0c4646d6bf22f04fcc74c8e70e760042012667c22df999c2b12f7e581602ebd0" exitCode=60
Nov 02 12:45:17.195973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.195922    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh" event={"ID":"cbb0f327-27f6-4bd4-99b5-85549548699e","Type":"ContainerDied","Data":"0c4646d6bf22f04fcc74c8e70e760042012667c22df999c2b12f7e581602ebd0"}
Nov 02 12:45:17.196264 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.196249    5879 scope.go:122] "RemoveContainer" containerID="0c4646d6bf22f04fcc74c8e70e760042012667c22df999c2b12f7e581602ebd0"
Nov 02 12:45:17.196492 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:17.196474    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh_odh-ai-gateway-infra(cbb0f327-27f6-4bd4-99b5-85549548699e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e"
Nov 02 12:45:17.197804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.197781    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm" event={"ID":"e442f069-a92e-4ab0-b936-0060f44b9e80","Type":"ContainerStarted","Data":"e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002"}
Nov 02 12:45:17.197894 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.197824    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm" event={"ID":"e442f069-a92e-4ab0-b936-0060f44b9e80","Type":"ContainerStarted","Data":"9047a8b75beab120bf28562d860754af6f5a9b5e37bacc0115fcffc355638a8c"}
Nov 02 12:45:17.197894 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.197866    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"
Nov 02 12:45:17.221589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.221563    5879 scope.go:122] "RemoveContainer" containerID="c65b09c54807914a8ccd54237426cd0b6214834e8dcd8345498ba2e78f91f715"
Nov 02 12:45:17.235961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.235935    5879 scope.go:122] "RemoveContainer" containerID="e19f1a7cb11a475028e16f16ad20bc82190be6eb927bfba943cc1f1428fa566d"
Nov 02 12:45:17.256781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.256754    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:17.432740 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.432707    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:17.447495 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.447450    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm" podStartSLOduration=1.447434571 podStartE2EDuration="1.447434571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:17.393550931 +0000 UTC m=+1063.500253851" watchObservedRunningTime="2026-11-02 12:45:17.447434571 +0000 UTC m=+1063.554137411"
Nov 02 12:45:17.451120 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.451095    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"]
Nov 02 12:45:17.455374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.455339    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-974cbcff9-gqzcj"]
Nov 02 12:45:17.504083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.497957    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:17.517542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.515118    5879 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 12:45:17.517542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.515180    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm" podUID="e442f069-a92e-4ab0-b936-0060f44b9e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 12:45:17.518700 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.518663    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"]
Nov 02 12:45:17.531692 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.531549    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-57df56c9cf-9rhzg"]
Nov 02 12:45:17.569874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:17.569844    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:18.014248 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.014213    5879 scope.go:122] "RemoveContainer" containerID="de43a4050805b37d8a5ead4303793d50bbd629de32d26ba488a362ad608d6e7a"
Nov 02 12:45:18.027154 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.027123    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277edcf7-7653-4f7f-b5af-958179c181c3" path="/var/lib/kubelet/pods/277edcf7-7653-4f7f-b5af-958179c181c3/volumes"
Nov 02 12:45:18.027693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.027672    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab41884-1345-483e-ab7d-362d118773bc" path="/var/lib/kubelet/pods/6ab41884-1345-483e-ab7d-362d118773bc/volumes"
Nov 02 12:45:18.219343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.219315    5879 generic.go:356] "Generic (PLEG): container finished" podID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" containerID="0b2c761404a31f7c9a0f3c0bb38f9817ac1631fab617cc435e558364782a4489" exitCode=60
Nov 02 12:45:18.219497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.219390    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9" event={"ID":"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09","Type":"ContainerDied","Data":"0b2c761404a31f7c9a0f3c0bb38f9817ac1631fab617cc435e558364782a4489"}
Nov 02 12:45:18.219497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.219459    5879 scope.go:122] "RemoveContainer" containerID="a961e4aad529c03375169461a041cca4f6871dd63cb8ebe019542b3a79ac968e"
Nov 02 12:45:18.219952 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.219937    5879 scope.go:122] "RemoveContainer" containerID="0b2c761404a31f7c9a0f3c0bb38f9817ac1631fab617cc435e558364782a4489"
Nov 02 12:45:18.220317 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:18.220293    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893725-96lz9_odh-ai-gateway-infra(e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09"
Nov 02 12:45:18.258804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.258773    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"]
Nov 02 12:45:18.316622 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.316534    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"]
Nov 02 12:45:18.333822 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.333779    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:18.348544 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.347882    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"]
Nov 02 12:45:18.461578 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.461543    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:18.513378 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.511793    5879 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 12:45:18.513378 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.511865    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm" podUID="e442f069-a92e-4ab0-b936-0060f44b9e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 12:45:18.568305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.568228    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"]
Nov 02 12:45:18.571908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.569361    5879 cpu_manager.go:414] "containerMap: removing container" podUID="277edcf7-7653-4f7f-b5af-958179c181c3" containerName="payload-pre-processing"
Nov 02 12:45:18.571908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.569384    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277edcf7-7653-4f7f-b5af-958179c181c3" containerName="payload-pre-processing"
Nov 02 12:45:18.571908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.569404    5879 cpu_manager.go:414] "containerMap: removing container" podUID="6ab41884-1345-483e-ab7d-362d118773bc" containerName="payload-processing"
Nov 02 12:45:18.571908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.569413    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab41884-1345-483e-ab7d-362d118773bc" containerName="payload-processing"
Nov 02 12:45:18.571908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.569577    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="277edcf7-7653-4f7f-b5af-958179c181c3" containerName="payload-pre-processing"
Nov 02 12:45:18.571908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.569592    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab41884-1345-483e-ab7d-362d118773bc" containerName="payload-processing"
Nov 02 12:45:18.579245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.579211    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"
Nov 02 12:45:18.636369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.628317    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-77cf70\""
Nov 02 12:45:18.636369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.629327    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-77cf70-dockercfg-x4g8h\""
Nov 02 12:45:18.656189 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.652936    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"]
Nov 02 12:45:18.763136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.763093    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569b20a0-e3a6-4c78-8551-a74dd77429d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k\" (UID: \"569b20a0-e3a6-4c78-8551-a74dd77429d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"
Nov 02 12:45:18.763318 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.763278    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsgdd\" (UniqueName: \"kubernetes.io/projected/569b20a0-e3a6-4c78-8551-a74dd77429d7-kube-api-access-fsgdd\") pod \"payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k\" (UID: \"569b20a0-e3a6-4c78-8551-a74dd77429d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"
Nov 02 12:45:18.767605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.767573    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:18.841607 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.836196    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"
Nov 02 12:45:18.865860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.864467    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsgdd\" (UniqueName: \"kubernetes.io/projected/569b20a0-e3a6-4c78-8551-a74dd77429d7-kube-api-access-fsgdd\") pod \"payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k\" (UID: \"569b20a0-e3a6-4c78-8551-a74dd77429d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"
Nov 02 12:45:18.865860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.864733    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569b20a0-e3a6-4c78-8551-a74dd77429d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k\" (UID: \"569b20a0-e3a6-4c78-8551-a74dd77429d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"
Nov 02 12:45:18.865860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.865748    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569b20a0-e3a6-4c78-8551-a74dd77429d7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k\" (UID: \"569b20a0-e3a6-4c78-8551-a74dd77429d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"
Nov 02 12:45:18.869443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.869362    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"
Nov 02 12:45:18.895017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.891051    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsgdd\" (UniqueName: \"kubernetes.io/projected/569b20a0-e3a6-4c78-8551-a74dd77429d7-kube-api-access-fsgdd\") pod \"payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k\" (UID: \"569b20a0-e3a6-4c78-8551-a74dd77429d7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"
Nov 02 12:45:18.966443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.966412    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb0f327-27f6-4bd4-99b5-85549548699e-kube-api-access-dj2k2\" (UniqueName: \"kubernetes.io/projected/cbb0f327-27f6-4bd4-99b5-85549548699e-kube-api-access-dj2k2\") pod \"cbb0f327-27f6-4bd4-99b5-85549548699e\" (UID: \"cbb0f327-27f6-4bd4-99b5-85549548699e\") "
Nov 02 12:45:18.966719 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.966705    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb0f327-27f6-4bd4-99b5-85549548699e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb0f327-27f6-4bd4-99b5-85549548699e-maas-api-service-ca\") pod \"cbb0f327-27f6-4bd4-99b5-85549548699e\" (UID: \"cbb0f327-27f6-4bd4-99b5-85549548699e\") "
Nov 02 12:45:18.966870 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.966855    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-maas-api-service-ca\") pod \"651a21a0-5eaf-4aa0-a3fc-d8760e824cba\" (UID: \"651a21a0-5eaf-4aa0-a3fc-d8760e824cba\") "
Nov 02 12:45:18.967132 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.967114    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-kube-api-access-6jzrx\" (UniqueName: \"kubernetes.io/projected/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-kube-api-access-6jzrx\") pod \"651a21a0-5eaf-4aa0-a3fc-d8760e824cba\" (UID: \"651a21a0-5eaf-4aa0-a3fc-d8760e824cba\") "
Nov 02 12:45:18.967781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.967750    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb0f327-27f6-4bd4-99b5-85549548699e-maas-api-service-ca" pod "cbb0f327-27f6-4bd4-99b5-85549548699e" (UID: "cbb0f327-27f6-4bd4-99b5-85549548699e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:18.968330 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.968311    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbb0f327-27f6-4bd4-99b5-85549548699e-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:18.968881 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.968854    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-maas-api-service-ca" pod "651a21a0-5eaf-4aa0-a3fc-d8760e824cba" (UID: "651a21a0-5eaf-4aa0-a3fc-d8760e824cba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:18.977414 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.977369    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb0f327-27f6-4bd4-99b5-85549548699e-kube-api-access-dj2k2" pod "cbb0f327-27f6-4bd4-99b5-85549548699e" (UID: "cbb0f327-27f6-4bd4-99b5-85549548699e"). InnerVolumeSpecName "kube-api-access-dj2k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:18.978678 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:18.978649    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-kube-api-access-6jzrx" pod "651a21a0-5eaf-4aa0-a3fc-d8760e824cba" (UID: "651a21a0-5eaf-4aa0-a3fc-d8760e824cba"). InnerVolumeSpecName "kube-api-access-6jzrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:19.070329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.070285    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jzrx\" (UniqueName: \"kubernetes.io/projected/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-kube-api-access-6jzrx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:19.070775 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.070335    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj2k2\" (UniqueName: \"kubernetes.io/projected/cbb0f327-27f6-4bd4-99b5-85549548699e-kube-api-access-dj2k2\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:19.070775 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.070350    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/651a21a0-5eaf-4aa0-a3fc-d8760e824cba-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:19.076575 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.076487    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:19.168510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.168478    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"
Nov 02 12:45:19.240151 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.240110    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s" event={"ID":"651a21a0-5eaf-4aa0-a3fc-d8760e824cba","Type":"ContainerDied","Data":"0a3dc98d58fe3d5b8b6b8b721f552407e1116eac500bb623bdce44ed4cd64311"}
Nov 02 12:45:19.240243 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.240181    5879 scope.go:122] "RemoveContainer" containerID="b17d6ea73000ba58890989314b02a027f6a2590a2d1d1ddc4d7f11930415997a"
Nov 02 12:45:19.240325 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.240310    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"
Nov 02 12:45:19.245640 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.245609    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" event={"ID":"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc","Type":"ContainerStarted","Data":"a5ae14f78d45e332f3c06b08b7c0d8d1fb26b386a02e3da72247b6f76c5e6721"}
Nov 02 12:45:19.246064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.246047    5879 scope.go:122] "RemoveContainer" containerID="a5ae14f78d45e332f3c06b08b7c0d8d1fb26b386a02e3da72247b6f76c5e6721"
Nov 02 12:45:19.246405 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:19.246383    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm_odh-ai-gateway-infra(4b650e3c-7ad6-4eb9-8f0d-14206e9046dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc"
Nov 02 12:45:19.247624 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.247598    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"
Nov 02 12:45:19.247704 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.247658    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh" event={"ID":"cbb0f327-27f6-4bd4-99b5-85549548699e","Type":"ContainerDied","Data":"d845b5e51a9815e5207e6ad3bba924db33342d0074d2b48e65e859efde3177e4"}
Nov 02 12:45:19.343570 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.343467    5879 scope.go:122] "RemoveContainer" containerID="0c4646d6bf22f04fcc74c8e70e760042012667c22df999c2b12f7e581602ebd0"
Nov 02 12:45:19.443071 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.441983    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"]
Nov 02 12:45:19.462604 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.447896    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893725-xjvxh"]
Nov 02 12:45:19.506020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.501805    5879 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 12:45:19.506020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.501862    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm" podUID="e442f069-a92e-4ab0-b936-0060f44b9e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 12:45:19.506251 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.506189    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"]
Nov 02 12:45:19.515877 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.511611    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893725-79w2s"]
Nov 02 12:45:19.644811 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.641799    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"]
Nov 02 12:45:19.678169 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.677817    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"
Nov 02 12:45:19.841617 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.841589    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-kube-api-access-c6j47\" (UniqueName: \"kubernetes.io/projected/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-kube-api-access-c6j47\") pod \"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09\" (UID: \"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09\") "
Nov 02 12:45:19.841795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.841661    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-maas-api-service-ca\") pod \"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09\" (UID: \"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09\") "
Nov 02 12:45:19.842050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.842028    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-maas-api-service-ca" pod "e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" (UID: "e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:19.842372 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.842358    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:19.843795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.843769    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-kube-api-access-c6j47" pod "e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" (UID: "e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09"). InnerVolumeSpecName "kube-api-access-c6j47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:19.906617 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.906564    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:19.944363 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:19.944332    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6j47\" (UniqueName: \"kubernetes.io/projected/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09-kube-api-access-c6j47\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:20.018417 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.018344    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" path="/var/lib/kubelet/pods/651a21a0-5eaf-4aa0-a3fc-d8760e824cba/volumes"
Nov 02 12:45:20.018766 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.018753    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e" path="/var/lib/kubelet/pods/cbb0f327-27f6-4bd4-99b5-85549548699e/volumes"
Nov 02 12:45:20.257947 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.257799    5879 generic.go:356] "Generic (PLEG): container finished" podID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" containerID="a5ae14f78d45e332f3c06b08b7c0d8d1fb26b386a02e3da72247b6f76c5e6721" exitCode=60
Nov 02 12:45:20.257947 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.257920    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" event={"ID":"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc","Type":"ContainerDied","Data":"a5ae14f78d45e332f3c06b08b7c0d8d1fb26b386a02e3da72247b6f76c5e6721"}
Nov 02 12:45:20.258486 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.258043    5879 scope.go:122] "RemoveContainer" containerID="de43a4050805b37d8a5ead4303793d50bbd629de32d26ba488a362ad608d6e7a"
Nov 02 12:45:20.258546 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.258510    5879 scope.go:122] "RemoveContainer" containerID="a5ae14f78d45e332f3c06b08b7c0d8d1fb26b386a02e3da72247b6f76c5e6721"
Nov 02 12:45:20.258857 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:20.258831    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm_odh-ai-gateway-infra(4b650e3c-7ad6-4eb9-8f0d-14206e9046dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc"
Nov 02 12:45:20.263891 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.263870    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k" event={"ID":"569b20a0-e3a6-4c78-8551-a74dd77429d7","Type":"ContainerStarted","Data":"d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64"}
Nov 02 12:45:20.264014 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.263916    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k" event={"ID":"569b20a0-e3a6-4c78-8551-a74dd77429d7","Type":"ContainerStarted","Data":"adf8b4804861fd7f5355015c67a2a4e82a7c612cda7848f44ce385a1db52a449"}
Nov 02 12:45:20.264014 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.263947    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"
Nov 02 12:45:20.269094 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.269045    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"
Nov 02 12:45:20.269210 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.269107    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9" event={"ID":"e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09","Type":"ContainerDied","Data":"cec30e7e14f808c9c4d1b8976fecd3571d14d2b0133b6d89f16e4d651c34e91c"}
Nov 02 12:45:20.276258 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.276239    5879 scope.go:122] "RemoveContainer" containerID="0b2c761404a31f7c9a0f3c0bb38f9817ac1631fab617cc435e558364782a4489"
Nov 02 12:45:20.396347 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.396286    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k" podStartSLOduration=2.396266101 podStartE2EDuration="2.396266101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:20.394904049 +0000 UTC m=+1066.501606902" watchObservedRunningTime="2026-11-02 12:45:20.396266101 +0000 UTC m=+1066.502968953"
Nov 02 12:45:20.503018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.499532    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:20.503018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.499932    5879 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 12:45:20.503018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.499978    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm" podUID="e442f069-a92e-4ab0-b936-0060f44b9e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 12:45:20.538070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.516915    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"]
Nov 02 12:45:20.626123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.626049    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"]
Nov 02 12:45:20.661460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:20.661332    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893725-96lz9"]
Nov 02 12:45:21.073161 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:21.073100    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:21.197016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:21.195439    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:21.498107 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:21.495116    5879 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 12:45:21.498107 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:21.495183    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm" podUID="e442f069-a92e-4ab0-b936-0060f44b9e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 12:45:21.553529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:21.553504    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"
Nov 02 12:45:21.652224 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:21.650985    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-maas-api-service-ca\") pod \"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc\" (UID: \"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc\") "
Nov 02 12:45:21.652224 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:21.651201    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-kube-api-access-6klhr\" (UniqueName: \"kubernetes.io/projected/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-kube-api-access-6klhr\") pod \"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc\" (UID: \"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc\") "
Nov 02 12:45:21.652897 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:21.652867    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-maas-api-service-ca" pod "4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" (UID: "4b650e3c-7ad6-4eb9-8f0d-14206e9046dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:45:21.672587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:21.672517    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-kube-api-access-6klhr" pod "4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" (UID: "4b650e3c-7ad6-4eb9-8f0d-14206e9046dc"). InnerVolumeSpecName "kube-api-access-6klhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:45:21.755038 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:21.753982    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6klhr\" (UniqueName: \"kubernetes.io/projected/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-kube-api-access-6klhr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:21.755038 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:21.754039    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:45:22.025508 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:22.025421    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" path="/var/lib/kubelet/pods/e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09/volumes"
Nov 02 12:45:22.323064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:22.322390    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm" event={"ID":"4b650e3c-7ad6-4eb9-8f0d-14206e9046dc","Type":"ContainerDied","Data":"c4a81f0893b5c0338ab4e06e18213b116d5f13d7ae370c0c38fb2a3f21683ef8"}
Nov 02 12:45:22.323064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:22.322464    5879 scope.go:122] "RemoveContainer" containerID="a5ae14f78d45e332f3c06b08b7c0d8d1fb26b386a02e3da72247b6f76c5e6721"
Nov 02 12:45:22.323064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:22.322575    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"
Nov 02 12:45:22.385020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:22.381297    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"]
Nov 02 12:45:22.411204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:22.411124    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cd1386cb-29893725-jqffm"]
Nov 02 12:45:22.494018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:22.491350    5879 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Nov 02 12:45:22.494018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:22.491419    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm" podUID="e442f069-a92e-4ab0-b936-0060f44b9e80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Nov 02 12:45:22.920014 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:22.917659    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:23.496809 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:23.496779    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:23.498190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:23.498170    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:23.498896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:23.498878    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:45:24.035278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.035244    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" path="/var/lib/kubelet/pods/4b650e3c-7ad6-4eb9-8f0d-14206e9046dc/volumes"
Nov 02 12:45:24.035724 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.035706    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"]
Nov 02 12:45:24.044176 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.044155    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6572f7e0-56m9b"]
Nov 02 12:45:24.080148 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.080116    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:24.234348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.234308    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:24.491103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.490066    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"]
Nov 02 12:45:24.491103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491039    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" containerName="cleanup"
Nov 02 12:45:24.491103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491056    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" containerName="cleanup"
Nov 02 12:45:24.491103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491076    5879 cpu_manager.go:414] "containerMap: removing container" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" containerName="cleanup"
Nov 02 12:45:24.491103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491085    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" containerName="cleanup"
Nov 02 12:45:24.491103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491102    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" containerName="cleanup"
Nov 02 12:45:24.491103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491109    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491121    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491128    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491144    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491153    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491169    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491176    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491188    5879 cpu_manager.go:414] "containerMap: removing container" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491194    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491206    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491212    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491232    5879 cpu_manager.go:414] "containerMap: removing container" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491239    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491274    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491280    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491448    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491460    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491471    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491483    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491495    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491505    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491516    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ffa5b3-83e8-4acc-9e31-8bc377c3aa09" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491526    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491535    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" containerName="cleanup"
Nov 02 12:45:24.492200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.491544    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="651a21a0-5eaf-4aa0-a3fc-d8760e824cba" containerName="cleanup"
Nov 02 12:45:24.498151 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.498122    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"]
Nov 02 12:45:24.498283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.498240    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"
Nov 02 12:45:24.533388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.518326    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0cf7e\""
Nov 02 12:45:24.533388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.531874    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0cf7e-dockercfg-r8jqc\""
Nov 02 12:45:24.626947 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.626917    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwmj9\" (UniqueName: \"kubernetes.io/projected/42d4b6ea-522f-4599-ab4c-c1605f4de134-kube-api-access-rwmj9\") pod \"payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd\" (UID: \"42d4b6ea-522f-4599-ab4c-c1605f4de134\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"
Nov 02 12:45:24.627142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.627009    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d4b6ea-522f-4599-ab4c-c1605f4de134-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd\" (UID: \"42d4b6ea-522f-4599-ab4c-c1605f4de134\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"
Nov 02 12:45:24.639770 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.639741    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"]
Nov 02 12:45:24.640580 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.640557    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e" containerName="cleanup"
Nov 02 12:45:24.640694 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.640581    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e" containerName="cleanup"
Nov 02 12:45:24.640694 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.640599    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" containerName="cleanup"
Nov 02 12:45:24.640694 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.640605    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" containerName="cleanup"
Nov 02 12:45:24.640810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.640725    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb0f327-27f6-4bd4-99b5-85549548699e" containerName="cleanup"
Nov 02 12:45:24.640810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.640739    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b650e3c-7ad6-4eb9-8f0d-14206e9046dc" containerName="cleanup"
Nov 02 12:45:24.644924 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.644903    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"
Nov 02 12:45:24.666330 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.666299    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"]
Nov 02 12:45:24.729902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.729863    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3889c386-b9da-45b0-9c9d-0300affa7b64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh\" (UID: \"3889c386-b9da-45b0-9c9d-0300affa7b64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"
Nov 02 12:45:24.730126 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.730072    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwmj9\" (UniqueName: \"kubernetes.io/projected/42d4b6ea-522f-4599-ab4c-c1605f4de134-kube-api-access-rwmj9\") pod \"payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd\" (UID: \"42d4b6ea-522f-4599-ab4c-c1605f4de134\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"
Nov 02 12:45:24.730126 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.730117    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsgzd\" (UniqueName: \"kubernetes.io/projected/3889c386-b9da-45b0-9c9d-0300affa7b64-kube-api-access-nsgzd\") pod \"payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh\" (UID: \"3889c386-b9da-45b0-9c9d-0300affa7b64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"
Nov 02 12:45:24.730250 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.730225    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d4b6ea-522f-4599-ab4c-c1605f4de134-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd\" (UID: \"42d4b6ea-522f-4599-ab4c-c1605f4de134\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"
Nov 02 12:45:24.730878 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.730857    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d4b6ea-522f-4599-ab4c-c1605f4de134-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd\" (UID: \"42d4b6ea-522f-4599-ab4c-c1605f4de134\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"
Nov 02 12:45:24.754650 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.754590    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwmj9\" (UniqueName: \"kubernetes.io/projected/42d4b6ea-522f-4599-ab4c-c1605f4de134-kube-api-access-rwmj9\") pod \"payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd\" (UID: \"42d4b6ea-522f-4599-ab4c-c1605f4de134\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"
Nov 02 12:45:24.824336 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.824301    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"
Nov 02 12:45:24.831392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.831360    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3889c386-b9da-45b0-9c9d-0300affa7b64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh\" (UID: \"3889c386-b9da-45b0-9c9d-0300affa7b64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"
Nov 02 12:45:24.831542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.831526    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsgzd\" (UniqueName: \"kubernetes.io/projected/3889c386-b9da-45b0-9c9d-0300affa7b64-kube-api-access-nsgzd\") pod \"payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh\" (UID: \"3889c386-b9da-45b0-9c9d-0300affa7b64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"
Nov 02 12:45:24.831973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.831955    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3889c386-b9da-45b0-9c9d-0300affa7b64-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh\" (UID: \"3889c386-b9da-45b0-9c9d-0300affa7b64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"
Nov 02 12:45:24.856435 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.856414    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsgzd\" (UniqueName: \"kubernetes.io/projected/3889c386-b9da-45b0-9c9d-0300affa7b64-kube-api-access-nsgzd\") pod \"payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh\" (UID: \"3889c386-b9da-45b0-9c9d-0300affa7b64\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"
Nov 02 12:45:24.961616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:24.961588    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"
Nov 02 12:45:25.130121 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:45:25.130092    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d4b6ea_522f_4599_ab4c_c1605f4de134.slice/crio-b56f8f21a873dc4087ef272039f840e8d6591b4f1410a72b438cd72d97720498 WatchSource:0}: Error finding container b56f8f21a873dc4087ef272039f840e8d6591b4f1410a72b438cd72d97720498: Status 404 returned error can't find the container with id b56f8f21a873dc4087ef272039f840e8d6591b4f1410a72b438cd72d97720498
Nov 02 12:45:25.131598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:25.131575    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"]
Nov 02 12:45:25.283546 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:45:25.283495    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3889c386_b9da_45b0_9c9d_0300affa7b64.slice/crio-7cbd22c09d68a12c0fb721b0af4a31b3b71a76b9eca7d0f4e95f56b4172a5e28 WatchSource:0}: Error finding container 7cbd22c09d68a12c0fb721b0af4a31b3b71a76b9eca7d0f4e95f56b4172a5e28: Status 404 returned error can't find the container with id 7cbd22c09d68a12c0fb721b0af4a31b3b71a76b9eca7d0f4e95f56b4172a5e28
Nov 02 12:45:25.284555 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:25.284534    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"]
Nov 02 12:45:25.317265 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:25.317242    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:25.355328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:25.355194    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd" event={"ID":"42d4b6ea-522f-4599-ab4c-c1605f4de134","Type":"ContainerStarted","Data":"b56f8f21a873dc4087ef272039f840e8d6591b4f1410a72b438cd72d97720498"}
Nov 02 12:45:25.357206 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:25.357178    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh" event={"ID":"3889c386-b9da-45b0-9c9d-0300affa7b64","Type":"ContainerStarted","Data":"7cbd22c09d68a12c0fb721b0af4a31b3b71a76b9eca7d0f4e95f56b4172a5e28"}
Nov 02 12:45:25.881193 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:25.881145    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:26.021831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:26.021800    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a831c41-4a83-4837-9c9f-a93452551bb0" path="/var/lib/kubelet/pods/2a831c41-4a83-4837-9c9f-a93452551bb0/volumes"
Nov 02 12:45:26.035515 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:26.035483    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:27.032536 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:27.032449    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"]
Nov 02 12:45:27.045477 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:27.045452    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d57be6-zc9ng"]
Nov 02 12:45:27.271856 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:27.271823    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:27.387478 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:27.387054    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd" event={"ID":"42d4b6ea-522f-4599-ab4c-c1605f4de134","Type":"ContainerStarted","Data":"a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939"}
Nov 02 12:45:27.387478 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:27.387183    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"
Nov 02 12:45:27.389242 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:27.389213    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh" event={"ID":"3889c386-b9da-45b0-9c9d-0300affa7b64","Type":"ContainerStarted","Data":"caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15"}
Nov 02 12:45:27.389395 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:27.389356    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"
Nov 02 12:45:27.433418 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:27.433368    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd" podStartSLOduration=1.9411825409999999 podStartE2EDuration="3.433353904s" totalImagesPullingTime="1.492171363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:27.430074354 +0000 UTC m=+1073.536777229" watchObservedRunningTime="2026-11-02 12:45:27.433353904 +0000 UTC m=+1073.540056755"
Nov 02 12:45:27.476206 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:27.476156    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh" podStartSLOduration=2.13616708 podStartE2EDuration="3.476141863s" totalImagesPullingTime="1.339974783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:27.473753641 +0000 UTC m=+1073.580456493" watchObservedRunningTime="2026-11-02 12:45:27.476141863 +0000 UTC m=+1073.582844714"
Nov 02 12:45:27.781320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:27.781235    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:27.902887 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:27.902856    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:28.049842 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:28.038252    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6" path="/var/lib/kubelet/pods/9bd1eabb-3ff8-4bab-8fcf-025e2d2e53d6/volumes"
Nov 02 12:45:28.971100 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:28.971060    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:30.457654 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:30.457622    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:31.297776 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:31.297748    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"
Nov 02 12:45:31.982560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:31.982527    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:33.683213 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:33.683184    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:33.806414 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:33.806382    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:34.084000 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:34.083915    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:34.467579 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:34.467540    5879 scope.go:122] "RemoveContainer" containerID="8deb9f9fafd244d6617433b055097223ac8b9a9997cfa0186017cb5f469456ed"
Nov 02 12:45:34.488655 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:34.488632    5879 scope.go:122] "RemoveContainer" containerID="5552696d7f9c546810569a735b490b40e809ea207e5dcf8722ae0ce6dfa1278a"
Nov 02 12:45:35.115378 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:35.115346    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:35.242938 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:35.242906    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:35.373145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:35.373050    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:37.011683 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:37.011647    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:38.033615 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:38.033585    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"]
Nov 02 12:45:38.046722 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:38.046698    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e84a277-7f2vg"]
Nov 02 12:45:38.401468 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:38.401442    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"
Nov 02 12:45:38.401618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:38.401506    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"
Nov 02 12:45:38.552182 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:38.552155    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:40.055040 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:40.055006    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab3d0ff-7fe2-4679-a45d-5aab696bdfff" path="/var/lib/kubelet/pods/4ab3d0ff-7fe2-4679-a45d-5aab696bdfff/volumes"
Nov 02 12:45:40.055476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:40.055457    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"]
Nov 02 12:45:40.059844 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:40.059820    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdc90d70-wpvqs"]
Nov 02 12:45:40.366236 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:40.366203    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:42.018670 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:42.018640    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08" path="/var/lib/kubelet/pods/1ae2fc15-5e71-4c3a-89c6-dbf59fb58a08/volumes"
Nov 02 12:45:42.344108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:42.344017    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:42.467205 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:42.467173    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:42.595273 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:42.595195    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:42.795149 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:42.795121    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:42.943849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:42.940464    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:44.241686 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:44.241610    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:44.820183 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:45:44.820146    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:45:45.849033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:45.848967    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:47.362413 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:47.362385    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:48.946648 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:48.946614    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:49.030405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:49.030374    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"]
Nov 02 12:45:49.046574 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:49.046541    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10e25d58-xpppq"]
Nov 02 12:45:50.017658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:50.017628    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67da26be-d09c-4c0d-99f1-b7f6a49dcc73" path="/var/lib/kubelet/pods/67da26be-d09c-4c0d-99f1-b7f6a49dcc73/volumes"
Nov 02 12:45:50.633780 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:50.633752    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:51.033062 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:51.032972    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"]
Nov 02 12:45:51.046156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:51.046128    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c3344ad-8x8bk"]
Nov 02 12:45:52.017135 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:52.017103    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7290a2ed-2d91-49af-a77e-bfb624c70ca7" path="/var/lib/kubelet/pods/7290a2ed-2d91-49af-a77e-bfb624c70ca7/volumes"
Nov 02 12:45:52.157111 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:52.157074    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:53.030925 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:53.030895    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"]
Nov 02 12:45:53.044304 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:53.044274    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f134cf3-vk5tg"]
Nov 02 12:45:53.479016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:53.478713    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:53.681143 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:53.681118    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:54.018579 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:54.018549    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbbfd85-88d0-4511-b66d-42d5c2fdfa75" path="/var/lib/kubelet/pods/5cbbfd85-88d0-4511-b66d-42d5c2fdfa75/volumes"
Nov 02 12:45:54.759855 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:54.759828    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:54.879885 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:54.879856    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:55.572727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:55.572693    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:55.783124 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:55.783085    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:57.055131 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:57.055098    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"]
Nov 02 12:45:57.266825 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:57.266790    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:57.400773 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:57.400718    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:58.558565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:58.558513    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:58.707449 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:58.707416    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:58.919253 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:58.919225    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"]
Nov 02 12:45:58.926882 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:58.926860    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:45:58.945665 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:58.945639    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"]
Nov 02 12:45:58.953047 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:58.952021    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e6ceda\""
Nov 02 12:45:58.953047 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:58.952111    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e6ceda-dockercfg-qpnww\""
Nov 02 12:45:58.967775 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:58.967734    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxqpf\" (UniqueName: \"kubernetes.io/projected/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-kube-api-access-nxqpf\") pod \"payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582\" (UID: \"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:45:58.969209 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:58.969179    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582\" (UID: \"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:45:59.071331 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.071298    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqpf\" (UniqueName: \"kubernetes.io/projected/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-kube-api-access-nxqpf\") pod \"payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582\" (UID: \"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:45:59.071486 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.071445    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582\" (UID: \"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:45:59.072147 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.072129    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582\" (UID: \"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:45:59.101395 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.101065    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"]
Nov 02 12:45:59.106922 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.106897    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"
Nov 02 12:45:59.113360 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.112791    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqpf\" (UniqueName: \"kubernetes.io/projected/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-kube-api-access-nxqpf\") pod \"payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582\" (UID: \"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:45:59.132783 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.132759    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"]
Nov 02 12:45:59.175122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.175030    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7tcz\" (UniqueName: \"kubernetes.io/projected/de061622-6a9e-49fc-b0fe-656f606c22ce-kube-api-access-m7tcz\") pod \"payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7\" (UID: \"de061622-6a9e-49fc-b0fe-656f606c22ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"
Nov 02 12:45:59.175294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.175169    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de061622-6a9e-49fc-b0fe-656f606c22ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7\" (UID: \"de061622-6a9e-49fc-b0fe-656f606c22ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"
Nov 02 12:45:59.245326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.245282    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:45:59.276639 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.276614    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de061622-6a9e-49fc-b0fe-656f606c22ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7\" (UID: \"de061622-6a9e-49fc-b0fe-656f606c22ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"
Nov 02 12:45:59.276879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.276863    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7tcz\" (UniqueName: \"kubernetes.io/projected/de061622-6a9e-49fc-b0fe-656f606c22ce-kube-api-access-m7tcz\") pod \"payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7\" (UID: \"de061622-6a9e-49fc-b0fe-656f606c22ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"
Nov 02 12:45:59.277344 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.277325    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de061622-6a9e-49fc-b0fe-656f606c22ce-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7\" (UID: \"de061622-6a9e-49fc-b0fe-656f606c22ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"
Nov 02 12:45:59.303901 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.303866    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7tcz\" (UniqueName: \"kubernetes.io/projected/de061622-6a9e-49fc-b0fe-656f606c22ce-kube-api-access-m7tcz\") pod \"payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7\" (UID: \"de061622-6a9e-49fc-b0fe-656f606c22ce\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"
Nov 02 12:45:59.452075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.451965    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"
Nov 02 12:45:59.575506 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.575445    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"]
Nov 02 12:45:59.671459 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.671430    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:45:59.679484 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.679462    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582" event={"ID":"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc","Type":"ContainerStarted","Data":"1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc"}
Nov 02 12:45:59.679611 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.679507    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582" event={"ID":"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc","Type":"ContainerStarted","Data":"b1932f7e7c734a871f9b5786851855e132cea514fae815964cbb3afc936dd4ae"}
Nov 02 12:45:59.679611 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.679532    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:45:59.727808 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.727712    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582" podStartSLOduration=1.7276988690000001 podStartE2EDuration="1.727698869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:45:59.726568147 +0000 UTC m=+1105.833270999" watchObservedRunningTime="2026-11-02 12:45:59.727698869 +0000 UTC m=+1105.834401720"
Nov 02 12:45:59.760141 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:45:59.760112    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde061622_6a9e_49fc_b0fe_656f606c22ce.slice/crio-3622c0f635f3d525cc5a831d0e3d84b6961cea26024c5ed060712b393d0745b1 WatchSource:0}: Error finding container 3622c0f635f3d525cc5a831d0e3d84b6961cea26024c5ed060712b393d0745b1: Status 404 returned error can't find the container with id 3622c0f635f3d525cc5a831d0e3d84b6961cea26024c5ed060712b393d0745b1
Nov 02 12:45:59.763164 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:45:59.763140    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"]
Nov 02 12:46:00.307474 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:00.307437    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:00.690975 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:00.690932    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7" event={"ID":"de061622-6a9e-49fc-b0fe-656f606c22ce","Type":"ContainerStarted","Data":"53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e"}
Nov 02 12:46:00.691485 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:00.691031    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7" event={"ID":"de061622-6a9e-49fc-b0fe-656f606c22ce","Type":"ContainerStarted","Data":"3622c0f635f3d525cc5a831d0e3d84b6961cea26024c5ed060712b393d0745b1"}
Nov 02 12:46:00.735691 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:00.735638    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7" podStartSLOduration=1.735626634 podStartE2EDuration="1.735626634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:00.734038614 +0000 UTC m=+1106.840741466" watchObservedRunningTime="2026-11-02 12:46:00.735626634 +0000 UTC m=+1106.842329542"
Nov 02 12:46:01.200905 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.200872    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"]
Nov 02 12:46:01.206639 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.206617    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.241342 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.241025    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"]
Nov 02 12:46:01.315348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.315312    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.315523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.315358    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be962747-84a6-4280-a965-2ed5b27f08b8-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.315523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.315494    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.315624 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.315556    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be962747-84a6-4280-a965-2ed5b27f08b8-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.315687 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.315661    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzzbz\" (UniqueName: \"kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-kube-api-access-zzzbz\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.315782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.315764    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.315846 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.315830    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.315902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.315871    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.315946 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.315933    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.416860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.416824    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.416860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.416858    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be962747-84a6-4280-a965-2ed5b27f08b8-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.417111 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.416920    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.417111 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.416973    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be962747-84a6-4280-a965-2ed5b27f08b8-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.417111 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.417025    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzzbz\" (UniqueName: \"kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-kube-api-access-zzzbz\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.417111 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.417089    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.417379 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.417352    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.417448 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.417408    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.417508 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.417452    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.417711 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.417689    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-data\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.417812 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.417789    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-certs\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.417894 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.417877    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.418071 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.418049    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be962747-84a6-4280-a965-2ed5b27f08b8-istiod-ca-cert\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.418295 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.418274    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-credential-socket\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.419737 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.419714    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be962747-84a6-4280-a965-2ed5b27f08b8-istio-podinfo\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.420293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.420270    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-envoy\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.442237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.442211    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-istio-token\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.442521 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.442500    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzzbz\" (UniqueName: \"kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-kube-api-access-zzzbz\") pod \"e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") " pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.526761 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.526671    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:01.698573 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.698538    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"
Nov 02 12:46:01.742051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.742021    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:01.851836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.851799    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"]
Nov 02 12:46:01.856366 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.856331    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:46:01.856470 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.856412    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:46:01.856470 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.856456    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:46:01.858516 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:01.858479    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:02.055631 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.055593    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k" podUID="569b20a0-e3a6-4c78-8551-a74dd77429d7" containerName="payload-processing" containerID="cri-o://d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64" gracePeriod=25
Nov 02 12:46:02.342218 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.342187    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"
Nov 02 12:46:02.439293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.433387    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569b20a0-e3a6-4c78-8551-a74dd77429d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569b20a0-e3a6-4c78-8551-a74dd77429d7-plugins-config-volume\") pod \"569b20a0-e3a6-4c78-8551-a74dd77429d7\" (UID: \"569b20a0-e3a6-4c78-8551-a74dd77429d7\") "
Nov 02 12:46:02.439293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.433574    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569b20a0-e3a6-4c78-8551-a74dd77429d7-kube-api-access-fsgdd\" (UniqueName: \"kubernetes.io/projected/569b20a0-e3a6-4c78-8551-a74dd77429d7-kube-api-access-fsgdd\") pod \"569b20a0-e3a6-4c78-8551-a74dd77429d7\" (UID: \"569b20a0-e3a6-4c78-8551-a74dd77429d7\") "
Nov 02 12:46:02.439293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.434240    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569b20a0-e3a6-4c78-8551-a74dd77429d7-plugins-config-volume" pod "569b20a0-e3a6-4c78-8551-a74dd77429d7" (UID: "569b20a0-e3a6-4c78-8551-a74dd77429d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:46:02.439293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.434427    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/569b20a0-e3a6-4c78-8551-a74dd77429d7-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:46:02.439293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.438445    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569b20a0-e3a6-4c78-8551-a74dd77429d7-kube-api-access-fsgdd" pod "569b20a0-e3a6-4c78-8551-a74dd77429d7" (UID: "569b20a0-e3a6-4c78-8551-a74dd77429d7"). InnerVolumeSpecName "kube-api-access-fsgdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:46:02.535815 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.535786    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsgdd\" (UniqueName: \"kubernetes.io/projected/569b20a0-e3a6-4c78-8551-a74dd77429d7-kube-api-access-fsgdd\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:46:02.731963 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.731872    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd" event={"ID":"be962747-84a6-4280-a965-2ed5b27f08b8","Type":"ContainerStarted","Data":"dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2"}
Nov 02 12:46:02.731963 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.731935    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd" event={"ID":"be962747-84a6-4280-a965-2ed5b27f08b8","Type":"ContainerStarted","Data":"61334d0ca05b7e3ffe8445c9aa05183fb875b9317a397d67c60beb437c6c6abf"}
Nov 02 12:46:02.733526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.733499    5879 generic.go:356] "Generic (PLEG): container finished" podID="569b20a0-e3a6-4c78-8551-a74dd77429d7" containerID="d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64" exitCode=0
Nov 02 12:46:02.733652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.733540    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"
Nov 02 12:46:02.733652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.733571    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k" event={"ID":"569b20a0-e3a6-4c78-8551-a74dd77429d7","Type":"ContainerDied","Data":"d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64"}
Nov 02 12:46:02.733652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.733618    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k" event={"ID":"569b20a0-e3a6-4c78-8551-a74dd77429d7","Type":"ContainerDied","Data":"adf8b4804861fd7f5355015c67a2a4e82a7c612cda7848f44ce385a1db52a449"}
Nov 02 12:46:02.733652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.733647    5879 scope.go:122] "RemoveContainer" containerID="d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64"
Nov 02 12:46:02.757512 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.757492    5879 scope.go:122] "RemoveContainer" containerID="d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64"
Nov 02 12:46:02.758915 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:46:02.758890    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64\": container with ID starting with d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64 not found: ID does not exist" containerID="d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64"
Nov 02 12:46:02.759061 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.758922    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64"} err="failed to get container status \"d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64\": rpc error: code = NotFound desc = could not find container \"d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64\": container with ID starting with d5929b9d1deaddfe33bd02ea8dc5d82dac3907d0dc0a07abed5ae780ba85bc64 not found: ID does not exist"
Nov 02 12:46:02.805015 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.803323    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd" podStartSLOduration=1.803303987 podStartE2EDuration="1.803303987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:02.778587903 +0000 UTC m=+1108.885290756" watchObservedRunningTime="2026-11-02 12:46:02.803303987 +0000 UTC m=+1108.910006839"
Nov 02 12:46:02.850173 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.850039    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"]
Nov 02 12:46:02.856776 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:02.856749    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-77cf70-587bc97bcf-rns5k"]
Nov 02 12:46:03.528044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:03.527317    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:03.544019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:03.534114    5879 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 12:46:03.544019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:03.534177    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd" podUID="be962747-84a6-4280-a965-2ed5b27f08b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 12:46:03.694026 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:03.691662    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:04.022823 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:04.022797    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569b20a0-e3a6-4c78-8551-a74dd77429d7" path="/var/lib/kubelet/pods/569b20a0-e3a6-4c78-8551-a74dd77429d7/volumes"
Nov 02 12:46:04.527299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:04.527262    5879 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 12:46:04.527474 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:04.527324    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd" podUID="be962747-84a6-4280-a965-2ed5b27f08b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 12:46:05.457218 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:05.457186    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:05.527510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:05.527474    5879 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 12:46:05.527658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:05.527545    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd" podUID="be962747-84a6-4280-a965-2ed5b27f08b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 12:46:06.030549 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:06.030520    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"]
Nov 02 12:46:06.043680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:06.043658    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaa85885-c86hr"]
Nov 02 12:46:06.527967 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:06.527927    5879 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 12:46:06.528454 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:06.527985    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd" podUID="be962747-84a6-4280-a965-2ed5b27f08b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 12:46:06.607631 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:06.607602    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:06.807072 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:06.804772    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:06.934449 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:06.934403    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:07.034194 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:07.034159    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"]
Nov 02 12:46:07.047378 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:07.047349    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0a4fb5b-c7tdj"]
Nov 02 12:46:07.197051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:07.194272    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:07.528022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:07.527913    5879 patch_prober.go:29] interesting pod/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body=
Nov 02 12:46:07.528022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:07.527968    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd" podUID="be962747-84a6-4280-a965-2ed5b27f08b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused"
Nov 02 12:46:08.017861 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:08.017828    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c68a59-59f8-47a4-a210-db61dacfda90" path="/var/lib/kubelet/pods/84c68a59-59f8-47a4-a210-db61dacfda90/volumes"
Nov 02 12:46:08.018333 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:08.018320    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4015f4c-88fe-48a4-a19a-918cb1333bfe" path="/var/lib/kubelet/pods/a4015f4c-88fe-48a4-a19a-918cb1333bfe/volumes"
Nov 02 12:46:08.534106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:08.534077    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:08.534694 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:08.534668    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:08.535710 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:08.535683    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:46:08.601238 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:08.601202    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:09.252163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:09.252050    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:10.693841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:10.693814    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:46:10.925259 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:10.925219    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:12.140685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:12.140653    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:12.281843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:12.281809    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:12.736437 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:12.736405    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"
Nov 02 12:46:13.257662 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:13.257621    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:14.290275 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:14.290239    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:14.753195 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:14.753129    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:14.822804 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:46:14.822763    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:46:14.895138 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:14.895104    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:15.046477 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:15.046280    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:16.460192 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:16.460153    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:17.383931 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:17.383902    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:17.514571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:17.514537    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:18.036320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:18.036287    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"]
Nov 02 12:46:18.049770 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:18.049741    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2386118d-tmww6"]
Nov 02 12:46:18.068225 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:18.068187    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:18.455584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:18.455554    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:18.857229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:18.857143    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:19.217028 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:19.216963    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:19.421246 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:19.421213    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:19.558501 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:19.558409    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:19.748422 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:19.748383    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:20.027468 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:20.027435    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff2474e-cdfe-45f2-9326-28084a6aed91" path="/var/lib/kubelet/pods/1ff2474e-cdfe-45f2-9326-28084a6aed91/volumes"
Nov 02 12:46:20.411651 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:20.411617    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:21.175757 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:21.175710    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:21.326575 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:21.326534    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:21.531360 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:21.531261    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:22.590984 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:22.590945    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:22.762134 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:22.762093    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:22.986016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:22.982566    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"]
Nov 02 12:46:23.005537 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.005496    5879 cpu_manager.go:414] "containerMap: removing container" podUID="569b20a0-e3a6-4c78-8551-a74dd77429d7" containerName="payload-processing"
Nov 02 12:46:23.005537 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.005534    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569b20a0-e3a6-4c78-8551-a74dd77429d7" containerName="payload-processing"
Nov 02 12:46:23.005872 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.005787    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="569b20a0-e3a6-4c78-8551-a74dd77429d7" containerName="payload-processing"
Nov 02 12:46:23.011028 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.010614    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"]
Nov 02 12:46:23.011028 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.010737    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"
Nov 02 12:46:23.041288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.041242    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-59060a-dockercfg-ktgls\""
Nov 02 12:46:23.102005 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.101970    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-59060a\""
Nov 02 12:46:23.129235 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.129201    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"]
Nov 02 12:46:23.133565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.133544    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"
Nov 02 12:46:23.141288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.141262    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmdzw\" (UniqueName: \"kubernetes.io/projected/449e1049-a8eb-44c0-9845-42d8b8d47603-kube-api-access-hmdzw\") pod \"payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv\" (UID: \"449e1049-a8eb-44c0-9845-42d8b8d47603\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"
Nov 02 12:46:23.141616 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.141600    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449e1049-a8eb-44c0-9845-42d8b8d47603-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv\" (UID: \"449e1049-a8eb-44c0-9845-42d8b8d47603\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"
Nov 02 12:46:23.154837 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.154814    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"]
Nov 02 12:46:23.243030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.242903    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb86z\" (UniqueName: \"kubernetes.io/projected/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-kube-api-access-mb86z\") pod \"payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw\" (UID: \"3a5e7064-f4d6-4ebd-a41c-0ee29d632763\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"
Nov 02 12:46:23.243205 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.243081    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449e1049-a8eb-44c0-9845-42d8b8d47603-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv\" (UID: \"449e1049-a8eb-44c0-9845-42d8b8d47603\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"
Nov 02 12:46:23.243205 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.243118    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw\" (UID: \"3a5e7064-f4d6-4ebd-a41c-0ee29d632763\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"
Nov 02 12:46:23.243205 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.243191    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmdzw\" (UniqueName: \"kubernetes.io/projected/449e1049-a8eb-44c0-9845-42d8b8d47603-kube-api-access-hmdzw\") pod \"payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv\" (UID: \"449e1049-a8eb-44c0-9845-42d8b8d47603\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"
Nov 02 12:46:23.243986 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.243959    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449e1049-a8eb-44c0-9845-42d8b8d47603-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv\" (UID: \"449e1049-a8eb-44c0-9845-42d8b8d47603\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"
Nov 02 12:46:23.268570 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.268540    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmdzw\" (UniqueName: \"kubernetes.io/projected/449e1049-a8eb-44c0-9845-42d8b8d47603-kube-api-access-hmdzw\") pod \"payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv\" (UID: \"449e1049-a8eb-44c0-9845-42d8b8d47603\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"
Nov 02 12:46:23.332653 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.332615    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"
Nov 02 12:46:23.345330 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.345289    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb86z\" (UniqueName: \"kubernetes.io/projected/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-kube-api-access-mb86z\") pod \"payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw\" (UID: \"3a5e7064-f4d6-4ebd-a41c-0ee29d632763\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"
Nov 02 12:46:23.345483 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.345398    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw\" (UID: \"3a5e7064-f4d6-4ebd-a41c-0ee29d632763\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"
Nov 02 12:46:23.346276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.346255    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw\" (UID: \"3a5e7064-f4d6-4ebd-a41c-0ee29d632763\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"
Nov 02 12:46:23.374576 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.372607    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb86z\" (UniqueName: \"kubernetes.io/projected/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-kube-api-access-mb86z\") pod \"payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw\" (UID: \"3a5e7064-f4d6-4ebd-a41c-0ee29d632763\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"
Nov 02 12:46:23.447565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.447297    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"
Nov 02 12:46:23.673809 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:46:23.673775    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449e1049_a8eb_44c0_9845_42d8b8d47603.slice/crio-0363841e338c6d6b6f22ad9556eb7ebaae4800d6c25e9ec2d8235599274be203 WatchSource:0}: Error finding container 0363841e338c6d6b6f22ad9556eb7ebaae4800d6c25e9ec2d8235599274be203: Status 404 returned error can't find the container with id 0363841e338c6d6b6f22ad9556eb7ebaae4800d6c25e9ec2d8235599274be203
Nov 02 12:46:23.674268 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.673809    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"]
Nov 02 12:46:23.693650 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.693625    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:23.785959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.785882    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"]
Nov 02 12:46:23.786254 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:46:23.786226    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5e7064_f4d6_4ebd_a41c_0ee29d632763.slice/crio-b5cc2544a0897b72f6ef34cfb07d4c6e59cee1769a2e0c9f24ff5584e51f33bc WatchSource:0}: Error finding container b5cc2544a0897b72f6ef34cfb07d4c6e59cee1769a2e0c9f24ff5584e51f33bc: Status 404 returned error can't find the container with id b5cc2544a0897b72f6ef34cfb07d4c6e59cee1769a2e0c9f24ff5584e51f33bc
Nov 02 12:46:23.908185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.907827    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:23.970570 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.970523    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw" event={"ID":"3a5e7064-f4d6-4ebd-a41c-0ee29d632763","Type":"ContainerStarted","Data":"b5cc2544a0897b72f6ef34cfb07d4c6e59cee1769a2e0c9f24ff5584e51f33bc"}
Nov 02 12:46:23.972440 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.972413    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv" event={"ID":"449e1049-a8eb-44c0-9845-42d8b8d47603","Type":"ContainerStarted","Data":"22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0"}
Nov 02 12:46:23.972593 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.972457    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv" event={"ID":"449e1049-a8eb-44c0-9845-42d8b8d47603","Type":"ContainerStarted","Data":"0363841e338c6d6b6f22ad9556eb7ebaae4800d6c25e9ec2d8235599274be203"}
Nov 02 12:46:23.973387 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:23.973368    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"
Nov 02 12:46:24.034491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:24.034419    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv" podStartSLOduration=2.034398829 podStartE2EDuration="2.034398829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:24.022085777 +0000 UTC m=+1130.128788629" watchObservedRunningTime="2026-11-02 12:46:24.034398829 +0000 UTC m=+1130.141101686"
Nov 02 12:46:24.980887 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:24.980847    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw" event={"ID":"3a5e7064-f4d6-4ebd-a41c-0ee29d632763","Type":"ContainerStarted","Data":"71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd"}
Nov 02 12:46:24.981361 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:24.981028    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"
Nov 02 12:46:25.025824 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:25.025760    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw" podStartSLOduration=2.025740839 podStartE2EDuration="2.025740839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:46:25.024425511 +0000 UTC m=+1131.131128386" watchObservedRunningTime="2026-11-02 12:46:25.025740839 +0000 UTC m=+1131.132443691"
Nov 02 12:46:25.403659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:25.403613    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:25.530511 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:25.530466    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:27.052114 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:27.052085    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:28.034406 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:28.034275    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"]
Nov 02 12:46:28.052397 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:28.049756    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2aae62d2-r4mv4"]
Nov 02 12:46:28.744565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:28.744522    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:29.301297 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:29.301263    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:29.761573 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:29.761536    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:29.966016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:29.965241    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:30.020594 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:30.020511    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232958a9-5f0e-4c72-9493-8f04d1ae6b50" path="/var/lib/kubelet/pods/232958a9-5f0e-4c72-9493-8f04d1ae6b50/volumes"
Nov 02 12:46:30.461156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:30.461126    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:32.583295 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:32.583262    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:33.312541 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:33.312468    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:33.438640 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:33.438601    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:34.350849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:34.350800    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:34.574422 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:34.574387    5879 scope.go:122] "RemoveContainer" containerID="7ea6558e477e9aa10f7680598f6d17d4cc46ce880df6b90d8808acb83e7b82f6"
Nov 02 12:46:34.584869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:34.584848    5879 scope.go:122] "RemoveContainer" containerID="2d7f4964acf9bfd577dcd3ad680710700909aa93cf0e588d388596f79fa6f4d6"
Nov 02 12:46:34.595014 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:34.594980    5879 scope.go:122] "RemoveContainer" containerID="cc3cc804e8a23da01acccbea85c7b9122bdeeb1f1f9cf9c2840e79563094e95f"
Nov 02 12:46:34.604203 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:34.604182    5879 scope.go:122] "RemoveContainer" containerID="937dc1767ceb90b37b8b323caeb074138e5df196c9b2a42eae754eef3ee85ee0"
Nov 02 12:46:34.613528 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:34.613510    5879 scope.go:122] "RemoveContainer" containerID="d7c0b7a6bb3313155642ff636bf33fe67aca04d4f19a85c232391bc0dbfab510"
Nov 02 12:46:35.020146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:35.020096    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:35.150644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:35.150606    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:35.310122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:35.304459    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:35.585109 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:35.584986    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:35.991054 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:35.990985    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"
Nov 02 12:46:35.991273 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:35.991220    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"
Nov 02 12:46:36.287334 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:36.287251    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:38.308661 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:38.308631    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:38.439088 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:38.439039    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:38.568284 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:38.568182    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:38.697188 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:38.697152    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:39.129735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:39.129700    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:39.307019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:39.303565    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:40.101381 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:40.101346    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:40.618792 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:40.618750    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:42.278913 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:42.278857    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:42.553675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:42.553571    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:43.449982 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:43.449948    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:43.584566 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:43.584529    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:44.527082 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:44.526987    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:44.661849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:44.661810    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:44.796756 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:44.796462    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:44.822469 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:46:44.822438    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:46:45.615662 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:45.615629    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:45.741239 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:45.741205    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:46.582306 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:46.582220    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:48.380006 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:48.379971    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:50.038531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:50.038489    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:51.729541 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:51.729502    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:53.494349 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:53.494306    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:55.608731 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:55.608699    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:55.733523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:55.733479    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:57.354154 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:57.354122    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:46:59.035314 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:46:59.035283    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:00.734636 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:00.734576    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:03.033470 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:03.033435    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:04.516350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:04.516307    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:04.588576 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:04.588531    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"]
Nov 02 12:47:04.588795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:04.588763    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh" podUID="3889c386-b9da-45b0-9c9d-0300affa7b64" containerName="payload-processing" containerID="cri-o://caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15" gracePeriod=30
Nov 02 12:47:04.610785 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:04.610739    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"]
Nov 02 12:47:04.611316 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:04.611260    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd" podUID="42d4b6ea-522f-4599-ab4c-c1605f4de134" containerName="payload-pre-processing" containerID="cri-o://a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939" gracePeriod=30
Nov 02 12:47:04.702870 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:04.702838    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:04.823217 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:04.823104    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:05.034019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.031703    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"
Nov 02 12:47:05.134978 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.134947    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"
Nov 02 12:47:05.196292 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.194341    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d4b6ea-522f-4599-ab4c-c1605f4de134-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d4b6ea-522f-4599-ab4c-c1605f4de134-plugins-config-volume\") pod \"42d4b6ea-522f-4599-ab4c-c1605f4de134\" (UID: \"42d4b6ea-522f-4599-ab4c-c1605f4de134\") "
Nov 02 12:47:05.196292 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.194736    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d4b6ea-522f-4599-ab4c-c1605f4de134-kube-api-access-rwmj9\" (UniqueName: \"kubernetes.io/projected/42d4b6ea-522f-4599-ab4c-c1605f4de134-kube-api-access-rwmj9\") pod \"42d4b6ea-522f-4599-ab4c-c1605f4de134\" (UID: \"42d4b6ea-522f-4599-ab4c-c1605f4de134\") "
Nov 02 12:47:05.198249 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.198216    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d4b6ea-522f-4599-ab4c-c1605f4de134-plugins-config-volume" pod "42d4b6ea-522f-4599-ab4c-c1605f4de134" (UID: "42d4b6ea-522f-4599-ab4c-c1605f4de134"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:05.201450 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.201417    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d4b6ea-522f-4599-ab4c-c1605f4de134-kube-api-access-rwmj9" pod "42d4b6ea-522f-4599-ab4c-c1605f4de134" (UID: "42d4b6ea-522f-4599-ab4c-c1605f4de134"). InnerVolumeSpecName "kube-api-access-rwmj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:05.297019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.296846    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3889c386-b9da-45b0-9c9d-0300affa7b64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3889c386-b9da-45b0-9c9d-0300affa7b64-plugins-config-volume\") pod \"3889c386-b9da-45b0-9c9d-0300affa7b64\" (UID: \"3889c386-b9da-45b0-9c9d-0300affa7b64\") "
Nov 02 12:47:05.297019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.296970    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3889c386-b9da-45b0-9c9d-0300affa7b64-kube-api-access-nsgzd\" (UniqueName: \"kubernetes.io/projected/3889c386-b9da-45b0-9c9d-0300affa7b64-kube-api-access-nsgzd\") pod \"3889c386-b9da-45b0-9c9d-0300affa7b64\" (UID: \"3889c386-b9da-45b0-9c9d-0300affa7b64\") "
Nov 02 12:47:05.297799 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.297768    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3889c386-b9da-45b0-9c9d-0300affa7b64-plugins-config-volume" pod "3889c386-b9da-45b0-9c9d-0300affa7b64" (UID: "3889c386-b9da-45b0-9c9d-0300affa7b64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:05.297929 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.297844    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwmj9\" (UniqueName: \"kubernetes.io/projected/42d4b6ea-522f-4599-ab4c-c1605f4de134-kube-api-access-rwmj9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:05.297929 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.297866    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42d4b6ea-522f-4599-ab4c-c1605f4de134-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:05.300233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.300199    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3889c386-b9da-45b0-9c9d-0300affa7b64-kube-api-access-nsgzd" pod "3889c386-b9da-45b0-9c9d-0300affa7b64" (UID: "3889c386-b9da-45b0-9c9d-0300affa7b64"). InnerVolumeSpecName "kube-api-access-nsgzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:05.330185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.330097    5879 generic.go:356] "Generic (PLEG): container finished" podID="42d4b6ea-522f-4599-ab4c-c1605f4de134" containerID="a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939" exitCode=0
Nov 02 12:47:05.330185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.330145    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"
Nov 02 12:47:05.330452 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.330187    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd" event={"ID":"42d4b6ea-522f-4599-ab4c-c1605f4de134","Type":"ContainerDied","Data":"a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939"}
Nov 02 12:47:05.330452 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.330248    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd" event={"ID":"42d4b6ea-522f-4599-ab4c-c1605f4de134","Type":"ContainerDied","Data":"b56f8f21a873dc4087ef272039f840e8d6591b4f1410a72b438cd72d97720498"}
Nov 02 12:47:05.330452 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.330278    5879 scope.go:122] "RemoveContainer" containerID="a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939"
Nov 02 12:47:05.331765 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.331745    5879 generic.go:356] "Generic (PLEG): container finished" podID="3889c386-b9da-45b0-9c9d-0300affa7b64" containerID="caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15" exitCode=0
Nov 02 12:47:05.331889 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.331794    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"
Nov 02 12:47:05.331889 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.331815    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh" event={"ID":"3889c386-b9da-45b0-9c9d-0300affa7b64","Type":"ContainerDied","Data":"caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15"}
Nov 02 12:47:05.331889 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.331845    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh" event={"ID":"3889c386-b9da-45b0-9c9d-0300affa7b64","Type":"ContainerDied","Data":"7cbd22c09d68a12c0fb721b0af4a31b3b71a76b9eca7d0f4e95f56b4172a5e28"}
Nov 02 12:47:05.341322 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.341297    5879 scope.go:122] "RemoveContainer" containerID="a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939"
Nov 02 12:47:05.341619 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:47:05.341596    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939\": container with ID starting with a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939 not found: ID does not exist" containerID="a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939"
Nov 02 12:47:05.341705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.341628    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939"} err="failed to get container status \"a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939\": rpc error: code = NotFound desc = could not find container \"a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939\": container with ID starting with a0141b268f17376d7707860a9ea7ffe8c429dad418c02d5f94e6b4721c8cb939 not found: ID does not exist"
Nov 02 12:47:05.341705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.341647    5879 scope.go:122] "RemoveContainer" containerID="caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15"
Nov 02 12:47:05.352312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.352290    5879 scope.go:122] "RemoveContainer" containerID="caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15"
Nov 02 12:47:05.352629 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:47:05.352612    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15\": container with ID starting with caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15 not found: ID does not exist" containerID="caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15"
Nov 02 12:47:05.352706 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.352637    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15"} err="failed to get container status \"caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15\": rpc error: code = NotFound desc = could not find container \"caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15\": container with ID starting with caa9729625a7385abc091095fb188dc7302612c07ef14627153bde02d647fb15 not found: ID does not exist"
Nov 02 12:47:05.387299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.387269    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"]
Nov 02 12:47:05.392576 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.392548    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0cf7e-7f48c94948-2n9fh"]
Nov 02 12:47:05.399488 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.399462    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3889c386-b9da-45b0-9c9d-0300affa7b64-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:05.399488 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.399485    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsgzd\" (UniqueName: \"kubernetes.io/projected/3889c386-b9da-45b0-9c9d-0300affa7b64-kube-api-access-nsgzd\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:05.435503 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.434896    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"]
Nov 02 12:47:05.439050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.439021    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0cf7e-8457687959-2hczd"]
Nov 02 12:47:05.496247 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.496199    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:05.635209 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.635176    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:05.757606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.757575    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:05.902196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.902110    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"]
Nov 02 12:47:05.902379 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:05.902354    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" podUID="0a14733e-c82f-4cae-8378-f87260f4b338" containerName="istio-proxy" containerID="cri-o://7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859" gracePeriod=30
Nov 02 12:47:06.025772 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:06.025742    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3889c386-b9da-45b0-9c9d-0300affa7b64" path="/var/lib/kubelet/pods/3889c386-b9da-45b0-9c9d-0300affa7b64/volumes"
Nov 02 12:47:06.026293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:06.026276    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d4b6ea-522f-4599-ab4c-c1605f4de134" path="/var/lib/kubelet/pods/42d4b6ea-522f-4599-ab4c-c1605f4de134/volumes"
Nov 02 12:47:07.453700 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:07.453669    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:08.789285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:08.789251    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:09.031519 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:09.031484    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"]
Nov 02 12:47:09.044577 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:09.044502    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-496b16f1-9d56h"]
Nov 02 12:47:09.652488 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:09.652457    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:10.018316 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:10.018242    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0187974-f552-49ad-a0d2-752794a67267" path="/var/lib/kubelet/pods/c0187974-f552-49ad-a0d2-752794a67267/volumes"
Nov 02 12:47:10.271211 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:10.271093    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:10.394198 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:10.394171    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:10.518615 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:10.518571    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:11.306828 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.306482    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:47:11.390279 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.390244    5879 generic.go:356] "Generic (PLEG): container finished" podID="0a14733e-c82f-4cae-8378-f87260f4b338" containerID="7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859" exitCode=0
Nov 02 12:47:11.390545 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.390525    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" event={"ID":"0a14733e-c82f-4cae-8378-f87260f4b338","Type":"ContainerDied","Data":"7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859"}
Nov 02 12:47:11.390599 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.390586    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm" event={"ID":"0a14733e-c82f-4cae-8378-f87260f4b338","Type":"ContainerDied","Data":"d5a28f9f21775a1b6a20777ebe8bcb203c47accbd68060f4298d016000dbfd44"}
Nov 02 12:47:11.390646 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.390621    5879 scope.go:122] "RemoveContainer" containerID="7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859"
Nov 02 12:47:11.390783 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.390769    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"
Nov 02 12:47:11.403896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.403876    5879 scope.go:122] "RemoveContainer" containerID="7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859"
Nov 02 12:47:11.404266 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:47:11.404241    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859\": container with ID starting with 7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859 not found: ID does not exist" containerID="7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859"
Nov 02 12:47:11.404350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.404278    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859"} err="failed to get container status \"7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859\": rpc error: code = NotFound desc = could not find container \"7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859\": container with ID starting with 7d22c6d25b4d545205cdd4d7690d4c6cab81775b8df5eef6100c8cc0c5785859 not found: ID does not exist"
Nov 02 12:47:11.407798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.407772    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-envoy\") pod \"0a14733e-c82f-4cae-8378-f87260f4b338\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") "
Nov 02 12:47:11.407877 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.407850    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-kube-api-access-w9x4h\" (UniqueName: \"kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-kube-api-access-w9x4h\") pod \"0a14733e-c82f-4cae-8378-f87260f4b338\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") "
Nov 02 12:47:11.407958 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.407944    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-socket\") pod \"0a14733e-c82f-4cae-8378-f87260f4b338\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") "
Nov 02 12:47:11.408043 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.407975    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a14733e-c82f-4cae-8378-f87260f4b338-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a14733e-c82f-4cae-8378-f87260f4b338-istiod-ca-cert\") pod \"0a14733e-c82f-4cae-8378-f87260f4b338\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") "
Nov 02 12:47:11.408156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.408137    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-data\") pod \"0a14733e-c82f-4cae-8378-f87260f4b338\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") "
Nov 02 12:47:11.408206 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.408184    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-istio-token\" (UniqueName: \"kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-istio-token\") pod \"0a14733e-c82f-4cae-8378-f87260f4b338\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") "
Nov 02 12:47:11.408319 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.408302    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a14733e-c82f-4cae-8378-f87260f4b338-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a14733e-c82f-4cae-8378-f87260f4b338-istio-podinfo\") pod \"0a14733e-c82f-4cae-8378-f87260f4b338\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") "
Nov 02 12:47:11.408387 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.408334    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-certs\") pod \"0a14733e-c82f-4cae-8378-f87260f4b338\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") "
Nov 02 12:47:11.408438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.408402    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-credential-socket\") pod \"0a14733e-c82f-4cae-8378-f87260f4b338\" (UID: \"0a14733e-c82f-4cae-8378-f87260f4b338\") "
Nov 02 12:47:11.408534 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.408509    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a14733e-c82f-4cae-8378-f87260f4b338-istiod-ca-cert" pod "0a14733e-c82f-4cae-8378-f87260f4b338" (UID: "0a14733e-c82f-4cae-8378-f87260f4b338"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:11.409278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.408817    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-socket" pod "0a14733e-c82f-4cae-8378-f87260f4b338" (UID: "0a14733e-c82f-4cae-8378-f87260f4b338"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:11.409278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.408923    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-data" pod "0a14733e-c82f-4cae-8378-f87260f4b338" (UID: "0a14733e-c82f-4cae-8378-f87260f4b338"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:11.409278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.409050    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-credential-socket" pod "0a14733e-c82f-4cae-8378-f87260f4b338" (UID: "0a14733e-c82f-4cae-8378-f87260f4b338"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:11.409278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.409246    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-certs" pod "0a14733e-c82f-4cae-8378-f87260f4b338" (UID: "0a14733e-c82f-4cae-8378-f87260f4b338"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:11.409572 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.409329    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:11.409572 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.409348    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a14733e-c82f-4cae-8378-f87260f4b338-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:11.409572 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.409363    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:11.409572 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.409376    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:11.410841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.410814    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-envoy" pod "0a14733e-c82f-4cae-8378-f87260f4b338" (UID: "0a14733e-c82f-4cae-8378-f87260f4b338"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:11.410934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.410876    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-istio-token" pod "0a14733e-c82f-4cae-8378-f87260f4b338" (UID: "0a14733e-c82f-4cae-8378-f87260f4b338"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:11.412039 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.411975    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a14733e-c82f-4cae-8378-f87260f4b338-istio-podinfo" pod "0a14733e-c82f-4cae-8378-f87260f4b338" (UID: "0a14733e-c82f-4cae-8378-f87260f4b338"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:47:11.412510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.412474    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-kube-api-access-w9x4h" pod "0a14733e-c82f-4cae-8378-f87260f4b338" (UID: "0a14733e-c82f-4cae-8378-f87260f4b338"). InnerVolumeSpecName "kube-api-access-w9x4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:11.512648 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.512478    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:11.512648 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.512518    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a14733e-c82f-4cae-8378-f87260f4b338-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:11.512648 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.512533    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:11.512648 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.512545    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a14733e-c82f-4cae-8378-f87260f4b338-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:11.512648 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.512558    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9x4h\" (UniqueName: \"kubernetes.io/projected/0a14733e-c82f-4cae-8378-f87260f4b338-kube-api-access-w9x4h\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:11.561425 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.561389    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:11.747400 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.746838    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"]
Nov 02 12:47:11.757848 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:11.757812    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-lm6gm"]
Nov 02 12:47:12.017640 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:12.017608    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a14733e-c82f-4cae-8378-f87260f4b338" path="/var/lib/kubelet/pods/0a14733e-c82f-4cae-8378-f87260f4b338/volumes"
Nov 02 12:47:13.348537 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:13.348509    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:13.601729 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:13.601596    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:13.731298 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:13.731261    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:14.116204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:14.116172    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"]
Nov 02 12:47:14.116398 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:14.116380    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql" podUID="270ac0a3-1976-47e7-8879-22a2ae545377" containerName="istio-proxy" containerID="cri-o://6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca" gracePeriod=30
Nov 02 12:47:14.368091 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:14.368010    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:14.766850 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:14.766773    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:14.822475 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:47:14.822437    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:47:14.898575 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:14.898539    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"]
Nov 02 12:47:14.898768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:14.898743    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd" podUID="be962747-84a6-4280-a965-2ed5b27f08b8" containerName="istio-proxy" containerID="cri-o://dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2" gracePeriod=30
Nov 02 12:47:14.913967 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:14.913930    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"]
Nov 02 12:47:14.914214 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:14.914192    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm" podUID="e442f069-a92e-4ab0-b936-0060f44b9e80" containerName="istio-proxy" containerID="cri-o://e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002" gracePeriod=30
Nov 02 12:47:14.936506 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:14.936463    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:15.282449 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:15.282420    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:15.559333 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:15.559248    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:15.737398 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:15.737370    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:16.088278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:16.088238    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:16.674555 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:16.674522    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:17.222651 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:17.222615    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:19.327618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.327579    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:19.430677 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.430646    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:47:19.456518 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.456488    5879 generic.go:356] "Generic (PLEG): container finished" podID="270ac0a3-1976-47e7-8879-22a2ae545377" containerID="6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca" exitCode=0
Nov 02 12:47:19.456735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.456545    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"
Nov 02 12:47:19.456735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.456573    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql" event={"ID":"270ac0a3-1976-47e7-8879-22a2ae545377","Type":"ContainerDied","Data":"6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca"}
Nov 02 12:47:19.456735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.456641    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql" event={"ID":"270ac0a3-1976-47e7-8879-22a2ae545377","Type":"ContainerDied","Data":"8d83aeb20e98d53361f8dba771bce3b6eb1557bd289d126dcabc8e809794900f"}
Nov 02 12:47:19.456735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.456677    5879 scope.go:122] "RemoveContainer" containerID="6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca"
Nov 02 12:47:19.457359 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457334    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-socket\") pod \"270ac0a3-1976-47e7-8879-22a2ae545377\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") "
Nov 02 12:47:19.457501 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457482    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-credential-socket\") pod \"270ac0a3-1976-47e7-8879-22a2ae545377\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") "
Nov 02 12:47:19.457659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457533    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-certs\") pod \"270ac0a3-1976-47e7-8879-22a2ae545377\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") "
Nov 02 12:47:19.457659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457577    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/270ac0a3-1976-47e7-8879-22a2ae545377-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270ac0a3-1976-47e7-8879-22a2ae545377-istio-podinfo\") pod \"270ac0a3-1976-47e7-8879-22a2ae545377\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") "
Nov 02 12:47:19.457659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457632    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-socket" pod "270ac0a3-1976-47e7-8879-22a2ae545377" (UID: "270ac0a3-1976-47e7-8879-22a2ae545377"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:19.457805 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457688    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-data\") pod \"270ac0a3-1976-47e7-8879-22a2ae545377\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") "
Nov 02 12:47:19.457805 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457779    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/270ac0a3-1976-47e7-8879-22a2ae545377-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270ac0a3-1976-47e7-8879-22a2ae545377-istiod-ca-cert\") pod \"270ac0a3-1976-47e7-8879-22a2ae545377\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") "
Nov 02 12:47:19.457805 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457795    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-credential-socket" pod "270ac0a3-1976-47e7-8879-22a2ae545377" (UID: "270ac0a3-1976-47e7-8879-22a2ae545377"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:19.457960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457880    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-istio-token\" (UniqueName: \"kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-istio-token\") pod \"270ac0a3-1976-47e7-8879-22a2ae545377\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") "
Nov 02 12:47:19.457960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457878    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-certs" pod "270ac0a3-1976-47e7-8879-22a2ae545377" (UID: "270ac0a3-1976-47e7-8879-22a2ae545377"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:19.457960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457891    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-data" pod "270ac0a3-1976-47e7-8879-22a2ae545377" (UID: "270ac0a3-1976-47e7-8879-22a2ae545377"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:19.457960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457909    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-envoy\") pod \"270ac0a3-1976-47e7-8879-22a2ae545377\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") "
Nov 02 12:47:19.458228 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.457967    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-kube-api-access-d47c9\" (UniqueName: \"kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-kube-api-access-d47c9\") pod \"270ac0a3-1976-47e7-8879-22a2ae545377\" (UID: \"270ac0a3-1976-47e7-8879-22a2ae545377\") "
Nov 02 12:47:19.458228 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.458181    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270ac0a3-1976-47e7-8879-22a2ae545377-istiod-ca-cert" pod "270ac0a3-1976-47e7-8879-22a2ae545377" (UID: "270ac0a3-1976-47e7-8879-22a2ae545377"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:19.458641 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.458623    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:19.458727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.458647    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270ac0a3-1976-47e7-8879-22a2ae545377-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:19.458727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.458660    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:19.458727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.458675    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:19.458727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.458689    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:19.460841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.460812    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/270ac0a3-1976-47e7-8879-22a2ae545377-istio-podinfo" pod "270ac0a3-1976-47e7-8879-22a2ae545377" (UID: "270ac0a3-1976-47e7-8879-22a2ae545377"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:47:19.460959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.460939    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-envoy" pod "270ac0a3-1976-47e7-8879-22a2ae545377" (UID: "270ac0a3-1976-47e7-8879-22a2ae545377"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:19.461063 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.461025    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-istio-token" pod "270ac0a3-1976-47e7-8879-22a2ae545377" (UID: "270ac0a3-1976-47e7-8879-22a2ae545377"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:19.461107 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.461059    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-kube-api-access-d47c9" pod "270ac0a3-1976-47e7-8879-22a2ae545377" (UID: "270ac0a3-1976-47e7-8879-22a2ae545377"). InnerVolumeSpecName "kube-api-access-d47c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:19.497671 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.497457    5879 scope.go:122] "RemoveContainer" containerID="6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca"
Nov 02 12:47:19.499060 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:47:19.498641    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca\": container with ID starting with 6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca not found: ID does not exist" containerID="6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca"
Nov 02 12:47:19.499060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.498702    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca"} err="failed to get container status \"6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca\": rpc error: code = NotFound desc = could not find container \"6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca\": container with ID starting with 6f69459f01659d15d0db015530afd574fdd0010a76a5b3dc1a54913b884394ca not found: ID does not exist"
Nov 02 12:47:19.559871 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.559837    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:19.559871 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.559866    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270ac0a3-1976-47e7-8879-22a2ae545377-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:19.559871 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.559877    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d47c9\" (UniqueName: \"kubernetes.io/projected/270ac0a3-1976-47e7-8879-22a2ae545377-kube-api-access-d47c9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:19.560152 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.559887    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270ac0a3-1976-47e7-8879-22a2ae545377-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:19.809198 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.809167    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"]
Nov 02 12:47:19.815604 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:19.815573    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-kcxql"]
Nov 02 12:47:20.020045 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.019955    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270ac0a3-1976-47e7-8879-22a2ae545377" path="/var/lib/kubelet/pods/270ac0a3-1976-47e7-8879-22a2ae545377/volumes"
Nov 02 12:47:20.212368 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.212346    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:47:20.244432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.244411    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:47:20.268770 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.268747    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-data\") pod \"be962747-84a6-4280-a965-2ed5b27f08b8\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") "
Nov 02 12:47:20.268941 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.268781    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-data\") pod \"e442f069-a92e-4ab0-b936-0060f44b9e80\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269042    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-data" pod "be962747-84a6-4280-a965-2ed5b27f08b8" (UID: "be962747-84a6-4280-a965-2ed5b27f08b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269079    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-data" pod "e442f069-a92e-4ab0-b936-0060f44b9e80" (UID: "e442f069-a92e-4ab0-b936-0060f44b9e80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269120    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-kube-api-access-zzzbz\" (UniqueName: \"kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-kube-api-access-zzzbz\") pod \"be962747-84a6-4280-a965-2ed5b27f08b8\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269170    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-envoy\") pod \"e442f069-a92e-4ab0-b936-0060f44b9e80\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269218    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-socket\") pod \"be962747-84a6-4280-a965-2ed5b27f08b8\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269254    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-token\" (UniqueName: \"kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-token\") pod \"e442f069-a92e-4ab0-b936-0060f44b9e80\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269313    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-kube-api-access-bntxl\" (UniqueName: \"kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-kube-api-access-bntxl\") pod \"e442f069-a92e-4ab0-b936-0060f44b9e80\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269395    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-envoy\") pod \"be962747-84a6-4280-a965-2ed5b27f08b8\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269429    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-credential-socket\") pod \"be962747-84a6-4280-a965-2ed5b27f08b8\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269485    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e442f069-a92e-4ab0-b936-0060f44b9e80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e442f069-a92e-4ab0-b936-0060f44b9e80-istiod-ca-cert\") pod \"e442f069-a92e-4ab0-b936-0060f44b9e80\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269566    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-podinfo\") pod \"e442f069-a92e-4ab0-b936-0060f44b9e80\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269570    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-socket" pod "be962747-84a6-4280-a965-2ed5b27f08b8" (UID: "be962747-84a6-4280-a965-2ed5b27f08b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269757    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-credential-socket" pod "be962747-84a6-4280-a965-2ed5b27f08b8" (UID: "be962747-84a6-4280-a965-2ed5b27f08b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269728    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-certs\") pod \"be962747-84a6-4280-a965-2ed5b27f08b8\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269817    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-certs\") pod \"e442f069-a92e-4ab0-b936-0060f44b9e80\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269853    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-istio-token\" (UniqueName: \"kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-istio-token\") pod \"be962747-84a6-4280-a965-2ed5b27f08b8\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.269899    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-credential-socket\") pod \"e442f069-a92e-4ab0-b936-0060f44b9e80\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.270029    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be962747-84a6-4280-a965-2ed5b27f08b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be962747-84a6-4280-a965-2ed5b27f08b8-istiod-ca-cert\") pod \"be962747-84a6-4280-a965-2ed5b27f08b8\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.270071    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-socket\") pod \"e442f069-a92e-4ab0-b936-0060f44b9e80\" (UID: \"e442f069-a92e-4ab0-b936-0060f44b9e80\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.270139    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be962747-84a6-4280-a965-2ed5b27f08b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be962747-84a6-4280-a965-2ed5b27f08b8-istio-podinfo\") pod \"be962747-84a6-4280-a965-2ed5b27f08b8\" (UID: \"be962747-84a6-4280-a965-2ed5b27f08b8\") "
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271018    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-certs" pod "be962747-84a6-4280-a965-2ed5b27f08b8" (UID: "be962747-84a6-4280-a965-2ed5b27f08b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271028    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e442f069-a92e-4ab0-b936-0060f44b9e80-istiod-ca-cert" pod "e442f069-a92e-4ab0-b936-0060f44b9e80" (UID: "e442f069-a92e-4ab0-b936-0060f44b9e80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271065    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-credential-socket" pod "e442f069-a92e-4ab0-b936-0060f44b9e80" (UID: "e442f069-a92e-4ab0-b936-0060f44b9e80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271233    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-socket" pod "e442f069-a92e-4ab0-b936-0060f44b9e80" (UID: "e442f069-a92e-4ab0-b936-0060f44b9e80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271331    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-certs" pod "e442f069-a92e-4ab0-b936-0060f44b9e80" (UID: "e442f069-a92e-4ab0-b936-0060f44b9e80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271714    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271733    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271745    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271759    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271774    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e442f069-a92e-4ab0-b936-0060f44b9e80-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271787    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271799    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271811    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.271824    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.273419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.272280    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be962747-84a6-4280-a965-2ed5b27f08b8-istiod-ca-cert" pod "be962747-84a6-4280-a965-2ed5b27f08b8" (UID: "be962747-84a6-4280-a965-2ed5b27f08b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:47:20.276181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.273798    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-kube-api-access-zzzbz" pod "be962747-84a6-4280-a965-2ed5b27f08b8" (UID: "be962747-84a6-4280-a965-2ed5b27f08b8"). InnerVolumeSpecName "kube-api-access-zzzbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:20.276181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.274024    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-istio-token" pod "be962747-84a6-4280-a965-2ed5b27f08b8" (UID: "be962747-84a6-4280-a965-2ed5b27f08b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:20.276181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.275049    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-podinfo" pod "e442f069-a92e-4ab0-b936-0060f44b9e80" (UID: "e442f069-a92e-4ab0-b936-0060f44b9e80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:47:20.276181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.275421    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-token" pod "e442f069-a92e-4ab0-b936-0060f44b9e80" (UID: "e442f069-a92e-4ab0-b936-0060f44b9e80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:20.276181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.275478    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-envoy" pod "be962747-84a6-4280-a965-2ed5b27f08b8" (UID: "be962747-84a6-4280-a965-2ed5b27f08b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.276181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.276069    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be962747-84a6-4280-a965-2ed5b27f08b8-istio-podinfo" pod "be962747-84a6-4280-a965-2ed5b27f08b8" (UID: "be962747-84a6-4280-a965-2ed5b27f08b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:47:20.276666 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.276581    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-envoy" pod "e442f069-a92e-4ab0-b936-0060f44b9e80" (UID: "e442f069-a92e-4ab0-b936-0060f44b9e80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:47:20.277899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.277877    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-kube-api-access-bntxl" pod "e442f069-a92e-4ab0-b936-0060f44b9e80" (UID: "e442f069-a92e-4ab0-b936-0060f44b9e80"). InnerVolumeSpecName "kube-api-access-bntxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:47:20.373642 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.373611    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.373642 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.373635    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.373642 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.373645    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be962747-84a6-4280-a965-2ed5b27f08b8-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.373642 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.373652    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be962747-84a6-4280-a965-2ed5b27f08b8-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.374103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.373660    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzzbz\" (UniqueName: \"kubernetes.io/projected/be962747-84a6-4280-a965-2ed5b27f08b8-kube-api-access-zzzbz\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.374103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.373668    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.374103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.373676    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.374103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.373683    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bntxl\" (UniqueName: \"kubernetes.io/projected/e442f069-a92e-4ab0-b936-0060f44b9e80-kube-api-access-bntxl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.374103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.373691    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be962747-84a6-4280-a965-2ed5b27f08b8-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:47:20.466735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.466704    5879 generic.go:356] "Generic (PLEG): container finished" podID="be962747-84a6-4280-a965-2ed5b27f08b8" containerID="dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2" exitCode=0
Nov 02 12:47:20.466946 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.466751    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"
Nov 02 12:47:20.466946 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.466794    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd" event={"ID":"be962747-84a6-4280-a965-2ed5b27f08b8","Type":"ContainerDied","Data":"dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2"}
Nov 02 12:47:20.466946 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.466848    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd" event={"ID":"be962747-84a6-4280-a965-2ed5b27f08b8","Type":"ContainerDied","Data":"61334d0ca05b7e3ffe8445c9aa05183fb875b9317a397d67c60beb437c6c6abf"}
Nov 02 12:47:20.466946 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.466877    5879 scope.go:122] "RemoveContainer" containerID="dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2"
Nov 02 12:47:20.468467 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.468404    5879 generic.go:356] "Generic (PLEG): container finished" podID="e442f069-a92e-4ab0-b936-0060f44b9e80" containerID="e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002" exitCode=0
Nov 02 12:47:20.468467 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.468460    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"
Nov 02 12:47:20.468708 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.468437    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm" event={"ID":"e442f069-a92e-4ab0-b936-0060f44b9e80","Type":"ContainerDied","Data":"e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002"}
Nov 02 12:47:20.468708 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.468526    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm" event={"ID":"e442f069-a92e-4ab0-b936-0060f44b9e80","Type":"ContainerDied","Data":"9047a8b75beab120bf28562d860754af6f5a9b5e37bacc0115fcffc355638a8c"}
Nov 02 12:47:20.487924 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.487892    5879 scope.go:122] "RemoveContainer" containerID="dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2"
Nov 02 12:47:20.488690 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:47:20.488658    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2\": container with ID starting with dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2 not found: ID does not exist" containerID="dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2"
Nov 02 12:47:20.489232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.488834    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2"} err="failed to get container status \"dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2\": rpc error: code = NotFound desc = could not find container \"dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2\": container with ID starting with dbd4b65db9b251fe740485c518f7aa528b0f8f71610633bdacb626f38c12a1d2 not found: ID does not exist"
Nov 02 12:47:20.489232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.488868    5879 scope.go:122] "RemoveContainer" containerID="e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002"
Nov 02 12:47:20.504662 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.504637    5879 scope.go:122] "RemoveContainer" containerID="e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002"
Nov 02 12:47:20.505052 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:47:20.505029    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002\": container with ID starting with e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002 not found: ID does not exist" containerID="e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002"
Nov 02 12:47:20.505167 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.505058    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002"} err="failed to get container status \"e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002\": rpc error: code = NotFound desc = could not find container \"e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002\": container with ID starting with e26ab7be79629ea810f89d2126d05039a9f63a1c262f7a36acfed2ce80f23002 not found: ID does not exist"
Nov 02 12:47:20.546535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.546446    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"]
Nov 02 12:47:20.550239 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.550215    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-q92sm"]
Nov 02 12:47:20.585841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.585809    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"]
Nov 02 12:47:20.590174 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:20.590148    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f0cf7e-openshift-default-86c8ff5b45-hw4gd"]
Nov 02 12:47:21.197045 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:21.196982    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:21.349383 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:21.349344    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:21.842396 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:21.842363    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:22.017822 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:22.017793    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be962747-84a6-4280-a965-2ed5b27f08b8" path="/var/lib/kubelet/pods/be962747-84a6-4280-a965-2ed5b27f08b8/volumes"
Nov 02 12:47:22.018331 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:22.018306    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e442f069-a92e-4ab0-b936-0060f44b9e80" path="/var/lib/kubelet/pods/e442f069-a92e-4ab0-b936-0060f44b9e80/volumes"
Nov 02 12:47:23.044520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:23.044490    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"]
Nov 02 12:47:23.057933 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:23.057899    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930d4880-drqxd"]
Nov 02 12:47:23.104340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:23.104302    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:23.238916 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:23.238874    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:23.966837 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:23.966783    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:24.018482 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:24.018450    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f7bc99-e37d-40c0-8948-aca0afb569de" path="/var/lib/kubelet/pods/93f7bc99-e37d-40c0-8948-aca0afb569de/volumes"
Nov 02 12:47:24.132665 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:24.132635    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:25.742490 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:25.742452    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:27.707234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:27.707200    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:29.766301 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:29.766265    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:29.908557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:29.908509    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:30.256051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:30.256008    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:31.668755 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:31.668721    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:31.858523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:31.858451    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:33.101305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:33.101266    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:34.066062 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:34.066023    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"]
Nov 02 12:47:34.078417 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:34.078385    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a451697-26468"]
Nov 02 12:47:34.675078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:34.675038    5879 scope.go:122] "RemoveContainer" containerID="3b2e8c047c071a9abdfe424064a6779514629ad384a1f893e903e04dc0b28c21"
Nov 02 12:47:34.686985 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:34.686932    5879 scope.go:122] "RemoveContainer" containerID="2d11e6f0d2481ab95056a80111213f68be571a23bd56acb29c22cf1fe66d3600"
Nov 02 12:47:34.699061 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:34.698926    5879 scope.go:122] "RemoveContainer" containerID="e9cb49b282427b3b7c2c455bed1bcb0fa2d4acad42332386f0406653f41208f5"
Nov 02 12:47:34.709195 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:34.709167    5879 scope.go:122] "RemoveContainer" containerID="6d9e192135d11ef6057df305196c7fd3295285fba3b54e56567d37c101552977"
Nov 02 12:47:34.719220 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:34.719198    5879 scope.go:122] "RemoveContainer" containerID="35d829d5b684a638006f359e9687504cc9385bf80029c964a4e584a970d4b4a8"
Nov 02 12:47:34.728968 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:34.728947    5879 scope.go:122] "RemoveContainer" containerID="1737e911df9ad778822c18f7d6cfb388c1c6530c43a76515f1033f8be8db3b11"
Nov 02 12:47:34.738828 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:34.738805    5879 scope.go:122] "RemoveContainer" containerID="249afb2059113c327b97bb2b2ae990e8cf13d5a5d74a4074a493df92a4012068"
Nov 02 12:47:34.748862 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:34.748834    5879 scope.go:122] "RemoveContainer" containerID="601f32d67358a0d1cd26036e2db730d41ddf6dec348ddab86ffad24766b4ee37"
Nov 02 12:47:34.758623 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:34.758600    5879 scope.go:122] "RemoveContainer" containerID="e283f12e0ba2985dcc1ff8c4e5f5313c75f26fcabf0fcb735fe22c9ae7435e26"
Nov 02 12:47:34.986920 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:34.986839    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:35.454091 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:35.454057    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:36.017702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:36.017669    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c5a86f-ba9c-43ee-8faa-4927990018da" path="/var/lib/kubelet/pods/84c5a86f-ba9c-43ee-8faa-4927990018da/volumes"
Nov 02 12:47:37.416341 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:37.416291    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:37.546843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:37.546810    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:38.979771 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:38.979737    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:39.860291 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:39.860236    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:39.982061 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:39.982018    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:40.130663 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:40.130630    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:40.635737 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:40.635700    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:42.278196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:42.278162    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:43.847272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:43.847235    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:44.037107 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:44.037078    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"]
Nov 02 12:47:44.049975 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:44.049949    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ac5c135-tmq5j"]
Nov 02 12:47:44.822603 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:47:44.822562    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:47:45.373063 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:45.372986    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:45.553838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:45.553801    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:45.715827 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:45.715746    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:46.018308 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:46.018230    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1568d745-66f3-49ee-ba2a-2d40ef44f708" path="/var/lib/kubelet/pods/1568d745-66f3-49ee-ba2a-2d40ef44f708/volumes"
Nov 02 12:47:46.316203 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:46.316120    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:46.445282 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:46.445246    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:48.392115 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:48.392080    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:48.769083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:48.769006    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:49.032748 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:49.032656    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"]
Nov 02 12:47:49.047667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:49.047631    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21c4c124-4mfgq"]
Nov 02 12:47:49.669867 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:49.669831    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:49.845712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:49.845670    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:50.027171 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:50.027084    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd718141-a4dd-49c3-b6e2-80a95e547bef" path="/var/lib/kubelet/pods/cd718141-a4dd-49c3-b6e2-80a95e547bef/volumes"
Nov 02 12:47:50.798351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:50.798315    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:52.147445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:52.147415    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:52.286449 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:52.286414    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:52.483016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:52.482909    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:52.931237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:52.931201    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:53.347839 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:53.347766    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:54.591439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:54.591396    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:55.117177 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:55.117141    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:55.647625 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:55.647591    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:55.837627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:55.833829    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:55.993544 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:55.993446    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:56.878680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:56.878645    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:47:59.273004 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:47:59.272936    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:00.055158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:00.055124    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:00.234972 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:00.234935    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:00.872204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:00.872173    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:01.075313 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:01.075281    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:02.003154 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:02.003122    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:02.140848 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:02.140810    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:04.042305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:04.042274    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:05.767644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:05.767612    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:07.007571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:07.007533    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:07.451142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:07.451108    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:07.587816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:07.587785    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:09.247603 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:09.247525    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:11.068561 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:11.068528    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:11.205246 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:11.205212    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:12.815891 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:12.815850    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:12.966348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:12.966313    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:14.824642 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:48:14.824599    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:48:14.925328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:14.925298    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:17.442069 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:17.442034    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:17.587828 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:17.587794    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:18.251389 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:18.251360    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:18.656827 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:18.656796    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:19.353922 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:19.353889    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:20.853747 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:20.853698    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:20.937687 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:20.937644    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"]
Nov 02 12:48:20.937943 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:20.937909    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7" podUID="de061622-6a9e-49fc-b0fe-656f606c22ce" containerName="payload-processing" containerID="cri-o://53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e" gracePeriod=30
Nov 02 12:48:20.969865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:20.969831    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"]
Nov 02 12:48:20.970334 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:20.970301    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582" podUID="e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc" containerName="payload-pre-processing" containerID="cri-o://1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc" gracePeriod=30
Nov 02 12:48:21.048228 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.048196    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:21.182696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.181726    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:21.344041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.339177    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"
Nov 02 12:48:21.381039 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.379709    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de061622-6a9e-49fc-b0fe-656f606c22ce-kube-api-access-m7tcz\" (UniqueName: \"kubernetes.io/projected/de061622-6a9e-49fc-b0fe-656f606c22ce-kube-api-access-m7tcz\") pod \"de061622-6a9e-49fc-b0fe-656f606c22ce\" (UID: \"de061622-6a9e-49fc-b0fe-656f606c22ce\") "
Nov 02 12:48:21.381039 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.379902    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de061622-6a9e-49fc-b0fe-656f606c22ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de061622-6a9e-49fc-b0fe-656f606c22ce-plugins-config-volume\") pod \"de061622-6a9e-49fc-b0fe-656f606c22ce\" (UID: \"de061622-6a9e-49fc-b0fe-656f606c22ce\") "
Nov 02 12:48:21.427146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.415802    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de061622-6a9e-49fc-b0fe-656f606c22ce-plugins-config-volume" pod "de061622-6a9e-49fc-b0fe-656f606c22ce" (UID: "de061622-6a9e-49fc-b0fe-656f606c22ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:21.427146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.421399    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de061622-6a9e-49fc-b0fe-656f606c22ce-kube-api-access-m7tcz" pod "de061622-6a9e-49fc-b0fe-656f606c22ce" (UID: "de061622-6a9e-49fc-b0fe-656f606c22ce"). InnerVolumeSpecName "kube-api-access-m7tcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:21.452119 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.447336    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:48:21.482445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.482329    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-plugins-config-volume\") pod \"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc\" (UID: \"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc\") "
Nov 02 12:48:21.483838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.482824    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-kube-api-access-nxqpf\" (UniqueName: \"kubernetes.io/projected/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-kube-api-access-nxqpf\") pod \"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc\" (UID: \"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc\") "
Nov 02 12:48:21.483838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.483453    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7tcz\" (UniqueName: \"kubernetes.io/projected/de061622-6a9e-49fc-b0fe-656f606c22ce-kube-api-access-m7tcz\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:48:21.483838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.483474    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de061622-6a9e-49fc-b0fe-656f606c22ce-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:48:21.484519 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.484457    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-plugins-config-volume" pod "e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc" (UID: "e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:21.486395 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.486368    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-kube-api-access-nxqpf" pod "e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc" (UID: "e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc"). InnerVolumeSpecName "kube-api-access-nxqpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:21.584884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.584852    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:48:21.584884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.584877    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxqpf\" (UniqueName: \"kubernetes.io/projected/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc-kube-api-access-nxqpf\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:48:21.694973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:21.694868    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:22.013195 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.013158    5879 generic.go:356] "Generic (PLEG): container finished" podID="de061622-6a9e-49fc-b0fe-656f606c22ce" containerID="53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e" exitCode=0
Nov 02 12:48:22.013695 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.013300    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"
Nov 02 12:48:22.015283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.015255    5879 generic.go:356] "Generic (PLEG): container finished" podID="e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc" containerID="1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc" exitCode=0
Nov 02 12:48:22.015473 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.015369    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:48:22.018329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.018306    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7" event={"ID":"de061622-6a9e-49fc-b0fe-656f606c22ce","Type":"ContainerDied","Data":"53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e"}
Nov 02 12:48:22.018461 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.018418    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7" event={"ID":"de061622-6a9e-49fc-b0fe-656f606c22ce","Type":"ContainerDied","Data":"3622c0f635f3d525cc5a831d0e3d84b6961cea26024c5ed060712b393d0745b1"}
Nov 02 12:48:22.018461 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.018443    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582" event={"ID":"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc","Type":"ContainerDied","Data":"1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc"}
Nov 02 12:48:22.018606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.018466    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582" event={"ID":"e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc","Type":"ContainerDied","Data":"b1932f7e7c734a871f9b5786851855e132cea514fae815964cbb3afc936dd4ae"}
Nov 02 12:48:22.019210 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.019187    5879 scope.go:122] "RemoveContainer" containerID="53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e"
Nov 02 12:48:22.032037 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.032020    5879 scope.go:122] "RemoveContainer" containerID="53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e"
Nov 02 12:48:22.032355 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:48:22.032339    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e\": container with ID starting with 53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e not found: ID does not exist" containerID="53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e"
Nov 02 12:48:22.032411 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.032362    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e"} err="failed to get container status \"53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e\": rpc error: code = NotFound desc = could not find container \"53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e\": container with ID starting with 53c542b59d5098cca0975e9649bb592f4c619b4e6bdb6c24361487b4aee65d7e not found: ID does not exist"
Nov 02 12:48:22.032411 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.032379    5879 scope.go:122] "RemoveContainer" containerID="1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc"
Nov 02 12:48:22.041967 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.041947    5879 scope.go:122] "RemoveContainer" containerID="1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc"
Nov 02 12:48:22.042260 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:48:22.042242    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc\": container with ID starting with 1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc not found: ID does not exist" containerID="1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc"
Nov 02 12:48:22.042336 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.042272    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc"} err="failed to get container status \"1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc\": rpc error: code = NotFound desc = could not find container \"1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc\": container with ID starting with 1f3e1ca36a9263e241f25d5c7e5ca9ee4de1e54fdb176646bfd6904c455241fc not found: ID does not exist"
Nov 02 12:48:22.069029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.068981    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"]
Nov 02 12:48:22.082579 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:22.082545    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6ceda-6dc8fb7994-zz6q7"]
Nov 02 12:48:23.863439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:23.863409    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:24.017401 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:24.017374    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de061622-6a9e-49fc-b0fe-656f606c22ce" path="/var/lib/kubelet/pods/de061622-6a9e-49fc-b0fe-656f606c22ce/volumes"
Nov 02 12:48:24.046218 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:24.046172    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:24.193487 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:24.193384    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:24.344277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:24.344245    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:24.495072 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:24.494552    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:25.055047 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:25.054983    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:25.451633 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:25.451604    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:26.327143 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:26.327112    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:26.485027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:26.484966    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:26.652307 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:26.652278    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:27.149267 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:27.149214    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:28.610902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:28.610863    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:28.681420 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:28.681380    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"]
Nov 02 12:48:28.681635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:28.681589    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw" podUID="3a5e7064-f4d6-4ebd-a41c-0ee29d632763" containerName="payload-processing" containerID="cri-o://71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd" gracePeriod=30
Nov 02 12:48:28.699029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:28.698897    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"]
Nov 02 12:48:28.699553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:28.699513    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv" podUID="449e1049-a8eb-44c0-9845-42d8b8d47603" containerName="payload-pre-processing" containerID="cri-o://22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0" gracePeriod=30
Nov 02 12:48:28.784587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:28.784552    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:28.925142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:28.924886    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:29.056040 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.055543    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"
Nov 02 12:48:29.063954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.062853    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:29.068484 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.068453    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"
Nov 02 12:48:29.073902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.073875    5879 generic.go:356] "Generic (PLEG): container finished" podID="3a5e7064-f4d6-4ebd-a41c-0ee29d632763" containerID="71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd" exitCode=0
Nov 02 12:48:29.074065 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.074046    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw" event={"ID":"3a5e7064-f4d6-4ebd-a41c-0ee29d632763","Type":"ContainerDied","Data":"71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd"}
Nov 02 12:48:29.074193 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.074111    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw" event={"ID":"3a5e7064-f4d6-4ebd-a41c-0ee29d632763","Type":"ContainerDied","Data":"b5cc2544a0897b72f6ef34cfb07d4c6e59cee1769a2e0c9f24ff5584e51f33bc"}
Nov 02 12:48:29.074193 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.074143    5879 scope.go:122] "RemoveContainer" containerID="71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd"
Nov 02 12:48:29.074329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.074294    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"
Nov 02 12:48:29.077458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.077411    5879 generic.go:356] "Generic (PLEG): container finished" podID="449e1049-a8eb-44c0-9845-42d8b8d47603" containerID="22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0" exitCode=0
Nov 02 12:48:29.077834 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.077649    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"
Nov 02 12:48:29.078184 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.078128    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv" event={"ID":"449e1049-a8eb-44c0-9845-42d8b8d47603","Type":"ContainerDied","Data":"22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0"}
Nov 02 12:48:29.078586 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.078524    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv" event={"ID":"449e1049-a8eb-44c0-9845-42d8b8d47603","Type":"ContainerDied","Data":"0363841e338c6d6b6f22ad9556eb7ebaae4800d6c25e9ec2d8235599274be203"}
Nov 02 12:48:29.086657 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.086636    5879 scope.go:122] "RemoveContainer" containerID="71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd"
Nov 02 12:48:29.087120 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:48:29.087095    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd\": container with ID starting with 71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd not found: ID does not exist" containerID="71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd"
Nov 02 12:48:29.087223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.087133    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd"} err="failed to get container status \"71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd\": rpc error: code = NotFound desc = could not find container \"71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd\": container with ID starting with 71e50b7f138a01e172f43f95815e6844bb30eea6bba4cd636897f117db76e1dd not found: ID does not exist"
Nov 02 12:48:29.087223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.087156    5879 scope.go:122] "RemoveContainer" containerID="22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0"
Nov 02 12:48:29.108028 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.100675    5879 scope.go:122] "RemoveContainer" containerID="22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0"
Nov 02 12:48:29.108028 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:48:29.101137    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0\": container with ID starting with 22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0 not found: ID does not exist" containerID="22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0"
Nov 02 12:48:29.108028 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.101172    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0"} err="failed to get container status \"22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0\": rpc error: code = NotFound desc = could not find container \"22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0\": container with ID starting with 22d923de22a60d2e87b4c26487f44888a53064c8d4f900a406bdfa939abdeef0 not found: ID does not exist"
Nov 02 12:48:29.288529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.270017    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/449e1049-a8eb-44c0-9845-42d8b8d47603-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449e1049-a8eb-44c0-9845-42d8b8d47603-plugins-config-volume\") pod \"449e1049-a8eb-44c0-9845-42d8b8d47603\" (UID: \"449e1049-a8eb-44c0-9845-42d8b8d47603\") "
Nov 02 12:48:29.288529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.270216    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-kube-api-access-mb86z\" (UniqueName: \"kubernetes.io/projected/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-kube-api-access-mb86z\") pod \"3a5e7064-f4d6-4ebd-a41c-0ee29d632763\" (UID: \"3a5e7064-f4d6-4ebd-a41c-0ee29d632763\") "
Nov 02 12:48:29.288529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.271050    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/449e1049-a8eb-44c0-9845-42d8b8d47603-kube-api-access-hmdzw\" (UniqueName: \"kubernetes.io/projected/449e1049-a8eb-44c0-9845-42d8b8d47603-kube-api-access-hmdzw\") pod \"449e1049-a8eb-44c0-9845-42d8b8d47603\" (UID: \"449e1049-a8eb-44c0-9845-42d8b8d47603\") "
Nov 02 12:48:29.288529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.271188    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-plugins-config-volume\") pod \"3a5e7064-f4d6-4ebd-a41c-0ee29d632763\" (UID: \"3a5e7064-f4d6-4ebd-a41c-0ee29d632763\") "
Nov 02 12:48:29.288529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.273799    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449e1049-a8eb-44c0-9845-42d8b8d47603-plugins-config-volume" pod "449e1049-a8eb-44c0-9845-42d8b8d47603" (UID: "449e1049-a8eb-44c0-9845-42d8b8d47603"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:29.288529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.287730    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449e1049-a8eb-44c0-9845-42d8b8d47603-kube-api-access-hmdzw" pod "449e1049-a8eb-44c0-9845-42d8b8d47603" (UID: "449e1049-a8eb-44c0-9845-42d8b8d47603"). InnerVolumeSpecName "kube-api-access-hmdzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:29.288529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.288444    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-plugins-config-volume" pod "3a5e7064-f4d6-4ebd-a41c-0ee29d632763" (UID: "3a5e7064-f4d6-4ebd-a41c-0ee29d632763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:48:29.306093 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.294542    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-kube-api-access-mb86z" pod "3a5e7064-f4d6-4ebd-a41c-0ee29d632763" (UID: "3a5e7064-f4d6-4ebd-a41c-0ee29d632763"). InnerVolumeSpecName "kube-api-access-mb86z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:48:29.374539 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.374113    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.374539 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.374163    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/449e1049-a8eb-44c0-9845-42d8b8d47603-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.374539 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.374179    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb86z\" (UniqueName: \"kubernetes.io/projected/3a5e7064-f4d6-4ebd-a41c-0ee29d632763-kube-api-access-mb86z\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.374539 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.374191    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmdzw\" (UniqueName: \"kubernetes.io/projected/449e1049-a8eb-44c0-9845-42d8b8d47603-kube-api-access-hmdzw\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:48:29.472028 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.468052    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"]
Nov 02 12:48:29.476032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.475589    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-59060a-7954596fdc-tlvvv"]
Nov 02 12:48:29.502116 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.502085    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"]
Nov 02 12:48:29.506183 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.506151    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-59060a-56fc94776-w9pkw"]
Nov 02 12:48:29.886742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:29.886630    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:30.017335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:30.017258    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a5e7064-f4d6-4ebd-a41c-0ee29d632763" path="/var/lib/kubelet/pods/3a5e7064-f4d6-4ebd-a41c-0ee29d632763/volumes"
Nov 02 12:48:30.017640 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:30.017628    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="449e1049-a8eb-44c0-9845-42d8b8d47603" path="/var/lib/kubelet/pods/449e1049-a8eb-44c0-9845-42d8b8d47603/volumes"
Nov 02 12:48:31.660333 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:31.660293    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:33.197582 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:33.197546    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:33.365817 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:33.365787    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:33.514628 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:33.514540    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:33.659858 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:33.659825    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:34.129936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:34.129017    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:34.465423 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:34.465310    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:34.610023 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:34.609501    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:34.857694 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:34.857604    5879 scope.go:122] "RemoveContainer" containerID="887b2e31e3787634ae2951298c5b9c3353f56ae386190018e5f006188ea9c85d"
Nov 02 12:48:34.868133 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:34.868110    5879 scope.go:122] "RemoveContainer" containerID="baec456dd7457abe8f6f10cf773a48b9875f0cf205f56e69813df082834342e7"
Nov 02 12:48:34.877557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:34.877534    5879 scope.go:122] "RemoveContainer" containerID="ab447e75dc3447295c0c7ec42d338abd8d9c79cab762f0841bc3265d3b9d716c"
Nov 02 12:48:35.244459 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:35.244429    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:35.378455 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:35.378421    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:35.574170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:35.574088    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:35.755027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:35.754955    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:35.916049 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:35.915776    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:36.467884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:36.467844    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:37.897508 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:37.897470    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:39.774270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:39.774235    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:41.706933 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:41.706895    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:41.857910 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:41.857873    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:42.360313 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:42.360282    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:42.499818 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:42.499765    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:43.553440 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:43.553367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:44.476692 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:44.476658    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:44.821966 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:48:44.821879    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:48:46.366185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:46.366152    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:47.849763 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:47.849722    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:48.124534 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:48.124501    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:48.266380 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:48.266341    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:48.647173 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:48.647120    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:49.263802 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:49.263759    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:49.409553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:49.409493    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:50.056954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:50.056916    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:50.194106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:50.194071    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:50.376675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:50.376443    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:50.516079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:50.516043    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:50.755076 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:50.749932    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:51.070442 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:51.070357    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:51.991709 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:51.991672    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:52.027564 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:52.027524    5879 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pode2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pode2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc] : Timed out while waiting for systemd to remove kubepods-burstable-pode2e4f5d3_489b_4d94_88a5_2a0f9e04f9dc.slice"
Nov 02 12:48:52.027739 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:48:52.027565    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods burstable pode2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc] : unable to destroy cgroup paths for cgroup [kubepods burstable pode2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc] : Timed out while waiting for systemd to remove kubepods-burstable-pode2e4f5d3_489b_4d94_88a5_2a0f9e04f9dc.slice" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582" podUID="e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc"
Nov 02 12:48:52.246851 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:52.246761    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"
Nov 02 12:48:52.347520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:52.347479    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"]
Nov 02 12:48:52.354857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:52.354827    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e6ceda-5f55dd9bc9-ck582"]
Nov 02 12:48:52.456760 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:52.456155    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:52.623144 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:52.621524    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:53.087789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:53.087155    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:54.020784 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:54.020748    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc" path="/var/lib/kubelet/pods/e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc/volumes"
Nov 02 12:48:54.967573 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:54.967531    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:55.782226 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:55.782183    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:55.944347 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:55.944314    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:56.083340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:56.083260    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:57.100478 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:57.100446    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:57.258908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:57.258876    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:57.564962 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:57.564930    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:57.723156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:57.723120    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:58.009019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:58.008362    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:59.072853 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:59.072822    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:59.196847 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:59.196816    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:48:59.905517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:48:59.905479    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:00.362751 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:00.362650    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:01.189604 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:01.189568    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:02.053600 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:02.053569    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:02.229247 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:02.229211    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:02.378027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:02.375238    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:02.931876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:02.931848    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:03.416909 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:03.416875    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:03.935083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:03.935054    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:04.485172 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:04.485124    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:04.622591 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:04.622538    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:05.015334 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:05.015302    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:06.884449 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:06.884412    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:07.036167 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:07.036131    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:07.175243 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:07.175152    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:08.032482 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:08.032445    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"]
Nov 02 12:49:08.046139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:08.046106    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b610da10-54dxc"]
Nov 02 12:49:08.693948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:08.693918    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:08.828748 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:08.828720    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:08.966693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:08.966606    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:09.318469 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:09.318384    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:09.452117 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:09.452076    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:10.018075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:10.018045    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2946fc9e-74f5-40ce-ae78-ac18fbb6a102" path="/var/lib/kubelet/pods/2946fc9e-74f5-40ce-ae78-ac18fbb6a102/volumes"
Nov 02 12:49:10.113454 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:10.113423    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:10.487703 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:10.487670    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:10.897682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:10.897638    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:11.434216 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:11.434180    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:11.900430 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:11.900389    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:12.073510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:12.073475    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:12.761225 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:12.761190    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:14.030171 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:14.030139    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"]
Nov 02 12:49:14.043575 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:14.043549    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f86dbf6-p5d9w"]
Nov 02 12:49:14.417814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:14.417781    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:14.557742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:14.557702    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:14.821164 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:49:14.821073    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:49:15.950924 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:15.950894    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:16.021162 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:16.021127    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11c3ef5-d6b3-45f9-a495-3119c068787f" path="/var/lib/kubelet/pods/d11c3ef5-d6b3-45f9-a495-3119c068787f/volumes"
Nov 02 12:49:16.125479 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:16.124321    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:16.546281 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:16.546235    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:17.747829 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:17.747786    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:17.922888 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:17.922847    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:18.576384 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:18.576344    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:19.911126 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:19.911090    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:20.125375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:20.125346    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:21.066149 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:21.066110    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:22.771292 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:22.771255    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:22.916880 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:22.916844    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:24.559627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:24.559579    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:26.836325 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:26.836011    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:28.451022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:28.450974    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:28.577351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:28.577309    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:28.698308 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:28.698258    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:30.362612 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:30.362582    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:32.273128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:32.273094    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:34.069213 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:34.069166    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:34.940943 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:34.940909    5879 scope.go:122] "RemoveContainer" containerID="2c21653c7cd9e47e041df24f5be124383c8e0e290f01061643309425aebd88df"
Nov 02 12:49:34.950970 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:34.950948    5879 scope.go:122] "RemoveContainer" containerID="62e9e8b626eff29ad2e9a44f9119c97285c120620390267ba86b84a7865d1925"
Nov 02 12:49:35.080011 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.079963    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"]
Nov 02 12:49:35.081447 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.081416    5879 cpu_manager.go:414] "containerMap: removing container" podUID="42d4b6ea-522f-4599-ab4c-c1605f4de134" containerName="payload-pre-processing"
Nov 02 12:49:35.081617 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.081603    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d4b6ea-522f-4599-ab4c-c1605f4de134" containerName="payload-pre-processing"
Nov 02 12:49:35.081707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.081696    5879 cpu_manager.go:414] "containerMap: removing container" podUID="de061622-6a9e-49fc-b0fe-656f606c22ce" containerName="payload-processing"
Nov 02 12:49:35.081790 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.081779    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de061622-6a9e-49fc-b0fe-656f606c22ce" containerName="payload-processing"
Nov 02 12:49:35.081872 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.081863    5879 cpu_manager.go:414] "containerMap: removing container" podUID="0a14733e-c82f-4cae-8378-f87260f4b338" containerName="istio-proxy"
Nov 02 12:49:35.081942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.081932    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a14733e-c82f-4cae-8378-f87260f4b338" containerName="istio-proxy"
Nov 02 12:49:35.082037 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.082027    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3a5e7064-f4d6-4ebd-a41c-0ee29d632763" containerName="payload-processing"
Nov 02 12:49:35.082117 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.082108    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a5e7064-f4d6-4ebd-a41c-0ee29d632763" containerName="payload-processing"
Nov 02 12:49:35.082203 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.082194    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3889c386-b9da-45b0-9c9d-0300affa7b64" containerName="payload-processing"
Nov 02 12:49:35.082272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.082262    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3889c386-b9da-45b0-9c9d-0300affa7b64" containerName="payload-processing"
Nov 02 12:49:35.082350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.082341    5879 cpu_manager.go:414] "containerMap: removing container" podUID="270ac0a3-1976-47e7-8879-22a2ae545377" containerName="istio-proxy"
Nov 02 12:49:35.082420 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.082410    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270ac0a3-1976-47e7-8879-22a2ae545377" containerName="istio-proxy"
Nov 02 12:49:35.082508 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.082497    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e442f069-a92e-4ab0-b936-0060f44b9e80" containerName="istio-proxy"
Nov 02 12:49:35.082581 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.082571    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e442f069-a92e-4ab0-b936-0060f44b9e80" containerName="istio-proxy"
Nov 02 12:49:35.082648 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.082639    5879 cpu_manager.go:414] "containerMap: removing container" podUID="be962747-84a6-4280-a965-2ed5b27f08b8" containerName="istio-proxy"
Nov 02 12:49:35.082723 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.082712    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be962747-84a6-4280-a965-2ed5b27f08b8" containerName="istio-proxy"
Nov 02 12:49:35.082838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.082826    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc" containerName="payload-pre-processing"
Nov 02 12:49:35.082918 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.082907    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc" containerName="payload-pre-processing"
Nov 02 12:49:35.083019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.083008    5879 cpu_manager.go:414] "containerMap: removing container" podUID="449e1049-a8eb-44c0-9845-42d8b8d47603" containerName="payload-pre-processing"
Nov 02 12:49:35.083104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.083094    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="449e1049-a8eb-44c0-9845-42d8b8d47603" containerName="payload-pre-processing"
Nov 02 12:49:35.083355 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.083344    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d4b6ea-522f-4599-ab4c-c1605f4de134" containerName="payload-pre-processing"
Nov 02 12:49:35.083434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.083425    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="de061622-6a9e-49fc-b0fe-656f606c22ce" containerName="payload-processing"
Nov 02 12:49:35.083517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.083508    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a14733e-c82f-4cae-8378-f87260f4b338" containerName="istio-proxy"
Nov 02 12:49:35.083590 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.083581    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a5e7064-f4d6-4ebd-a41c-0ee29d632763" containerName="payload-processing"
Nov 02 12:49:35.083660 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.083651    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3889c386-b9da-45b0-9c9d-0300affa7b64" containerName="payload-processing"
Nov 02 12:49:35.083728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.083719    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="270ac0a3-1976-47e7-8879-22a2ae545377" containerName="istio-proxy"
Nov 02 12:49:35.083796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.083788    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e442f069-a92e-4ab0-b936-0060f44b9e80" containerName="istio-proxy"
Nov 02 12:49:35.083858 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.083850    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="be962747-84a6-4280-a965-2ed5b27f08b8" containerName="istio-proxy"
Nov 02 12:49:35.083928 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.083920    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e4f5d3-489b-4d94-88a5-2a0f9e04f9dc" containerName="payload-pre-processing"
Nov 02 12:49:35.084015 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.084004    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="449e1049-a8eb-44c0-9845-42d8b8d47603" containerName="payload-pre-processing"
Nov 02 12:49:35.090316 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.090297    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.123341 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.121878    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"]
Nov 02 12:49:35.131526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.131133    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-729108\"/\"ipp-route-729108-kserve-self-signed-certs\""
Nov 02 12:49:35.131526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.131169    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-729108\"/\"openshift-service-ca.crt\""
Nov 02 12:49:35.131526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.131170    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-729108\"/\"kube-root-ca.crt\""
Nov 02 12:49:35.131526 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.131227    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-729108\"/\"default-dockercfg-l92t8\""
Nov 02 12:49:35.198360 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.198278    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:35.226580 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.226542    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-dshm\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.226758 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.226621    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjsp\" (UniqueName: \"kubernetes.io/projected/61768aba-42e4-4b0c-87e3-53c4e4e17c66-kube-api-access-nbjsp\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.226802 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.226756    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-model-cache\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.226928 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.226905    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tmp-dir\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.227032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.226981    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tls-certs\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.227101 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.227082    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-home\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.331567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.331523    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-model-cache\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.332518 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.332482    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tmp-dir\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.333097 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.333077    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tls-certs\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.333332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.333308    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-home\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.333621 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.333604    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-dshm\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.333808 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.333791    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjsp\" (UniqueName: \"kubernetes.io/projected/61768aba-42e4-4b0c-87e3-53c4e4e17c66-kube-api-access-nbjsp\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.337262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.332981    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tmp-dir\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.341847 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.341353    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-home\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.341847 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.332324    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-model-cache\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.375755 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.366768    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-dshm\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.375755 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.371745    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjsp\" (UniqueName: \"kubernetes.io/projected/61768aba-42e4-4b0c-87e3-53c4e4e17c66-kube-api-access-nbjsp\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.375755 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.372248    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tls-certs\") pod \"ipp-route-729108-kserve-7f65b4675c-w458s\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") " pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.438531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.438244    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:35.707137 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.707052    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:35.804382 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:49:35.804350    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61768aba_42e4_4b0c_87e3_53c4e4e17c66.slice/crio-01b69f5a63bee990358ca426a127df5ca6e1a4b30cd6e746c68534ffd4e0c58c WatchSource:0}: Error finding container 01b69f5a63bee990358ca426a127df5ca6e1a4b30cd6e746c68534ffd4e0c58c: Status 404 returned error can't find the container with id 01b69f5a63bee990358ca426a127df5ca6e1a4b30cd6e746c68534ffd4e0c58c
Nov 02 12:49:35.805424 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.805404    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"]
Nov 02 12:49:35.917973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:35.917934    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:36.603509 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:36.603476    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s" event={"ID":"61768aba-42e4-4b0c-87e3-53c4e4e17c66","Type":"ContainerStarted","Data":"314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037"}
Nov 02 12:49:36.603904 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:36.603521    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s" event={"ID":"61768aba-42e4-4b0c-87e3-53c4e4e17c66","Type":"ContainerStarted","Data":"01b69f5a63bee990358ca426a127df5ca6e1a4b30cd6e746c68534ffd4e0c58c"}
Nov 02 12:49:37.438939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:37.438894    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:37.601143 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:37.601107    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:39.460912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:39.460869    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:39.596841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:39.596791    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:39.731301 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:39.731228    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:41.353800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:41.353761    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:41.494783 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:41.494747    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:42.666633 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:42.666597    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:43.142067 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:43.141970    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:43.295058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:43.295022    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:43.439776 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:43.439527    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:44.821592 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:49:44.821554    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:49:45.050060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:45.050028    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:45.198018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:45.192205    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:45.439904 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:45.439870    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:45.452912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:45.452853    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:45.504607 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:45.504551    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s" podStartSLOduration=10.504531758 podStartE2EDuration="10.504531758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:36.658514471 +0000 UTC m=+1322.765217323" watchObservedRunningTime="2026-11-02 12:49:45.504531758 +0000 UTC m=+1331.611234610"
Nov 02 12:49:45.641161 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:45.641128    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:45.684100 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:45.684064    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:49:47.460351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:47.460312    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:47.594192 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:47.594153    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:49.221677 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.221641    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:49.374414 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.374378    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:49.377346 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.377311    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"]
Nov 02 12:49:49.382446 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.382424    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"
Nov 02 12:49:49.400934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.400910    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"]
Nov 02 12:49:49.411053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.411027    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9750c7-dockercfg-gclk9\""
Nov 02 12:49:49.411053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.411029    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9750c7\""
Nov 02 12:49:49.476843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.476757    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89k8j\" (UniqueName: \"kubernetes.io/projected/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-kube-api-access-89k8j\") pod \"payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs\" (UID: \"f6ad669c-85d5-4c25-9e69-9348efb2d1ca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"
Nov 02 12:49:49.477072 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.477029    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs\" (UID: \"f6ad669c-85d5-4c25-9e69-9348efb2d1ca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"
Nov 02 12:49:49.550675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.550636    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:49.600020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.579071    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k8j\" (UniqueName: \"kubernetes.io/projected/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-kube-api-access-89k8j\") pod \"payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs\" (UID: \"f6ad669c-85d5-4c25-9e69-9348efb2d1ca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"
Nov 02 12:49:49.601190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.601157    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs\" (UID: \"f6ad669c-85d5-4c25-9e69-9348efb2d1ca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"
Nov 02 12:49:49.602717 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.602690    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs\" (UID: \"f6ad669c-85d5-4c25-9e69-9348efb2d1ca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"
Nov 02 12:49:49.621301 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.621244    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89k8j\" (UniqueName: \"kubernetes.io/projected/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-kube-api-access-89k8j\") pod \"payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs\" (UID: \"f6ad669c-85d5-4c25-9e69-9348efb2d1ca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"
Nov 02 12:49:49.699026 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:49.698954    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"
Nov 02 12:49:50.067844 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:50.067817    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"]
Nov 02 12:49:50.719514 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:50.719471    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs" event={"ID":"f6ad669c-85d5-4c25-9e69-9348efb2d1ca","Type":"ContainerStarted","Data":"dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e"}
Nov 02 12:49:50.719965 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:50.719543    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs" event={"ID":"f6ad669c-85d5-4c25-9e69-9348efb2d1ca","Type":"ContainerStarted","Data":"45ebbdcb6ba6c22b4abf5653670ee5d87b0a2ea745a44147e89963e5fe9b00dc"}
Nov 02 12:49:50.719965 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:50.719579    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"
Nov 02 12:49:50.764726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:50.764662    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs" podStartSLOduration=1.764642942 podStartE2EDuration="1.764642942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:49:50.762017308 +0000 UTC m=+1336.868720160" watchObservedRunningTime="2026-11-02 12:49:50.764642942 +0000 UTC m=+1336.871345794"
Nov 02 12:49:52.082951 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:52.082909    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:52.584216 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:52.584182    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:53.853337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:53.853296    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:55.688386 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:55.688350    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:57.018301 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:57.018265    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:57.614680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:57.614616    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:49:57.759063 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:49:57.759028    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:00.708321 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:00.708289    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:00.840735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:00.840704    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:01.308906 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:01.308864    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:01.728633 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:01.728604    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"
Nov 02 12:50:03.187006 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:03.186950    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:03.426270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:03.426232    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:03.570416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:03.570331    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:04.320799 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:04.320753    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:05.058539 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.058506    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:05.196201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.196169    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:05.338204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.338116    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:05.357133 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.357106    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"]
Nov 02 12:50:05.361478 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.361457    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"
Nov 02 12:50:05.382722 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.382691    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"]
Nov 02 12:50:05.393122 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.393093    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-d3a281\""
Nov 02 12:50:05.393307 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.393165    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-d3a281-dockercfg-5glrg\""
Nov 02 12:50:05.459970 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.459931    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfb5287-27fe-483b-a7c3-45e6fe896c95-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx\" (UID: \"8dfb5287-27fe-483b-a7c3-45e6fe896c95\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"
Nov 02 12:50:05.460491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.460440    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p88vm\" (UniqueName: \"kubernetes.io/projected/8dfb5287-27fe-483b-a7c3-45e6fe896c95-kube-api-access-p88vm\") pod \"payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx\" (UID: \"8dfb5287-27fe-483b-a7c3-45e6fe896c95\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"
Nov 02 12:50:05.562622 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.562584    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p88vm\" (UniqueName: \"kubernetes.io/projected/8dfb5287-27fe-483b-a7c3-45e6fe896c95-kube-api-access-p88vm\") pod \"payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx\" (UID: \"8dfb5287-27fe-483b-a7c3-45e6fe896c95\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"
Nov 02 12:50:05.563034 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.562960    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfb5287-27fe-483b-a7c3-45e6fe896c95-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx\" (UID: \"8dfb5287-27fe-483b-a7c3-45e6fe896c95\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"
Nov 02 12:50:05.563784 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.563759    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfb5287-27fe-483b-a7c3-45e6fe896c95-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx\" (UID: \"8dfb5287-27fe-483b-a7c3-45e6fe896c95\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"
Nov 02 12:50:05.595112 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.595031    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p88vm\" (UniqueName: \"kubernetes.io/projected/8dfb5287-27fe-483b-a7c3-45e6fe896c95-kube-api-access-p88vm\") pod \"payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx\" (UID: \"8dfb5287-27fe-483b-a7c3-45e6fe896c95\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"
Nov 02 12:50:05.675170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.675135    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"
Nov 02 12:50:05.994104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:05.994075    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"]
Nov 02 12:50:06.845626 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:06.845582    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx" event={"ID":"8dfb5287-27fe-483b-a7c3-45e6fe896c95","Type":"ContainerStarted","Data":"8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c"}
Nov 02 12:50:06.846124 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:06.845646    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx" event={"ID":"8dfb5287-27fe-483b-a7c3-45e6fe896c95","Type":"ContainerStarted","Data":"510455f30a0945b2fc2ea2cefc087319ca6fb74e6765bb22d00188f30e5af4fa"}
Nov 02 12:50:06.846124 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:06.845676    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"
Nov 02 12:50:06.889224 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:06.889173    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx" podStartSLOduration=1.88915765 podStartE2EDuration="1.88915765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:06.887821377 +0000 UTC m=+1352.994524210" watchObservedRunningTime="2026-11-02 12:50:06.88915765 +0000 UTC m=+1352.995860506"
Nov 02 12:50:06.915405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:06.915360    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:08.745256 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:08.745207    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:10.680232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:10.680200    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:10.956700 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:10.956607    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:12.073442 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:12.073405    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:12.251931 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:12.251850    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:12.397614 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:12.397578    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:12.718597 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:12.718522    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:12.849208 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:12.849173    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:13.113561 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:13.113516    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:14.431053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:14.431022    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:14.825759 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:50:14.825679    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:50:16.104707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:16.104673    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:17.858137 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:17.858110    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"
Nov 02 12:50:18.254161 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:18.254127    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:19.864512 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:19.864480    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:21.534315 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:21.534265    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:22.979426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:22.979390    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:23.658138 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:23.658106    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:25.234342 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:25.234308    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:25.682356 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:25.682318    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:25.810500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:25.810471    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:26.362904 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:26.362867    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:26.729537 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:26.729487    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:26.904443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:26.904408    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:27.171742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:27.171708    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:28.552166 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:28.552131    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:28.948405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:28.948373    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"]
Nov 02 12:50:28.948737 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:28.948604    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy" containerID="cri-o://493c25b10bb68b4dbdc8952ea40b910eb30cb7685c2c865f7fe37c4ec705c568" gracePeriod=30
Nov 02 12:50:30.290810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:30.290767    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:31.396783 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:31.396743    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:31.546955 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:31.546912    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:31.940682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:31.940654    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"]
Nov 02 12:50:31.945385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:31.945365    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"
Nov 02 12:50:31.964613 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:31.964541    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"]
Nov 02 12:50:32.030603 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.030574    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-eb24df-dockercfg-d2gvt\""
Nov 02 12:50:32.039648 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.039620    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-eb24df\""
Nov 02 12:50:32.052915 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.052884    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:32.076468 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.076435    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwxs\" (UniqueName: \"kubernetes.io/projected/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-kube-api-access-2wwxs\") pod \"payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f\" (UID: \"46418b5e-3bb7-4f4a-b5e0-86dba55616d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"
Nov 02 12:50:32.076743 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.076726    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f\" (UID: \"46418b5e-3bb7-4f4a-b5e0-86dba55616d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"
Nov 02 12:50:32.178495 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.178455    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f\" (UID: \"46418b5e-3bb7-4f4a-b5e0-86dba55616d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"
Nov 02 12:50:32.178691 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.178573    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwxs\" (UniqueName: \"kubernetes.io/projected/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-kube-api-access-2wwxs\") pod \"payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f\" (UID: \"46418b5e-3bb7-4f4a-b5e0-86dba55616d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"
Nov 02 12:50:32.179217 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.179193    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f\" (UID: \"46418b5e-3bb7-4f4a-b5e0-86dba55616d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"
Nov 02 12:50:32.203261 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.203202    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwxs\" (UniqueName: \"kubernetes.io/projected/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-kube-api-access-2wwxs\") pod \"payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f\" (UID: \"46418b5e-3bb7-4f4a-b5e0-86dba55616d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"
Nov 02 12:50:32.260151 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.260113    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"
Nov 02 12:50:32.455425 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.455343    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:32.567382 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.567354    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"]
Nov 02 12:50:32.570086 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:50:32.570048    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46418b5e_3bb7_4f4a_b5e0_86dba55616d5.slice/crio-b0e56a77d54ec4c41cfd1349788cd9a55cfe82bec41be547e87463291eb7159a WatchSource:0}: Error finding container b0e56a77d54ec4c41cfd1349788cd9a55cfe82bec41be547e87463291eb7159a: Status 404 returned error can't find the container with id b0e56a77d54ec4c41cfd1349788cd9a55cfe82bec41be547e87463291eb7159a
Nov 02 12:50:32.632136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.632109    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"]
Nov 02 12:50:32.632342 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:32.632322    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" podUID="11648a63-a758-4f5e-be54-976d4e234612" containerName="istio-proxy" containerID="cri-o://67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750" gracePeriod=30
Nov 02 12:50:33.043796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:33.043751    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f" event={"ID":"46418b5e-3bb7-4f4a-b5e0-86dba55616d5","Type":"ContainerStarted","Data":"c49c17df416c32d5d649c88d21edc6c58557d5951612b2c2479212fd3a78eec4"}
Nov 02 12:50:33.044010 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:33.043830    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f" event={"ID":"46418b5e-3bb7-4f4a-b5e0-86dba55616d5","Type":"ContainerStarted","Data":"b0e56a77d54ec4c41cfd1349788cd9a55cfe82bec41be547e87463291eb7159a"}
Nov 02 12:50:33.044265 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:33.044244    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"
Nov 02 12:50:33.088758 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:33.088697    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f" podStartSLOduration=2.088684687 podStartE2EDuration="2.088684687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:33.087920768 +0000 UTC m=+1379.194623618" watchObservedRunningTime="2026-11-02 12:50:33.088684687 +0000 UTC m=+1379.195387537"
Nov 02 12:50:34.235337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.235304    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:50:34.277560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.277521    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.318296    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-token\" (UniqueName: \"kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-token\") pod \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") "
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.318338    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-envoy\") pod \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") "
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.318370    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-socket\") pod \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") "
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.318562    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-kube-api-access-jx7hs\" (UniqueName: \"kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-kube-api-access-jx7hs\") pod \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") "
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.318614    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-podinfo\") pod \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") "
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.318646    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-credential-socket\") pod \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") "
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.318720    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-data\") pod \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") "
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.318757    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-certs\") pod \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") "
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.318780    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istiod-ca-cert\") pod \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\" (UID: \"2844d7db-1a7a-4ebd-93c8-c49944bbb954\") "
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.319968    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-socket" pod "2844d7db-1a7a-4ebd-93c8-c49944bbb954" (UID: "2844d7db-1a7a-4ebd-93c8-c49944bbb954"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.320230    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-credential-socket" pod "2844d7db-1a7a-4ebd-93c8-c49944bbb954" (UID: "2844d7db-1a7a-4ebd-93c8-c49944bbb954"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.320452    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istiod-ca-cert" pod "2844d7db-1a7a-4ebd-93c8-c49944bbb954" (UID: "2844d7db-1a7a-4ebd-93c8-c49944bbb954"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.320682    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-data" pod "2844d7db-1a7a-4ebd-93c8-c49944bbb954" (UID: "2844d7db-1a7a-4ebd-93c8-c49944bbb954"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:34.324027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.320833    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-certs" pod "2844d7db-1a7a-4ebd-93c8-c49944bbb954" (UID: "2844d7db-1a7a-4ebd-93c8-c49944bbb954"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:34.352067 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.341980    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-podinfo" pod "2844d7db-1a7a-4ebd-93c8-c49944bbb954" (UID: "2844d7db-1a7a-4ebd-93c8-c49944bbb954"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:50:34.435019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.421281    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:34.435019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.421319    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:34.435019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.421335    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:34.435019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.421357    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:34.435019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.421372    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:34.435019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.421384    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:34.435744 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.435625    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-kube-api-access-jx7hs" pod "2844d7db-1a7a-4ebd-93c8-c49944bbb954" (UID: "2844d7db-1a7a-4ebd-93c8-c49944bbb954"). InnerVolumeSpecName "kube-api-access-jx7hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:34.448287 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.446599    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-token" pod "2844d7db-1a7a-4ebd-93c8-c49944bbb954" (UID: "2844d7db-1a7a-4ebd-93c8-c49944bbb954"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:34.448287 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.447326    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-envoy" pod "2844d7db-1a7a-4ebd-93c8-c49944bbb954" (UID: "2844d7db-1a7a-4ebd-93c8-c49944bbb954"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:34.498598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.498508    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:34.522497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.522464    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:34.522497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.522495    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2844d7db-1a7a-4ebd-93c8-c49944bbb954-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:34.522718 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.522507    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx7hs\" (UniqueName: \"kubernetes.io/projected/2844d7db-1a7a-4ebd-93c8-c49944bbb954-kube-api-access-jx7hs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:34.632215 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.632181    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:34.982363 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.982336    5879 scope.go:122] "RemoveContainer" containerID="1f9910d663715538f5cbf6a62495fa39b0b1a3f7677c3b061f4fe820d661fc37"
Nov 02 12:50:34.996163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:34.996137    5879 scope.go:122] "RemoveContainer" containerID="0c9969bda84a3469b2a9094fd937ae844d37a72dfae01adac6b0e0c0f9295410"
Nov 02 12:50:35.006615 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:35.006593    5879 scope.go:122] "RemoveContainer" containerID="431c8afe851c1e8e7b2f6a54130ef8ba3c764688e76fd84fc074e98e9a678190"
Nov 02 12:50:35.015378 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:35.015359    5879 scope.go:122] "RemoveContainer" containerID="493c25b10bb68b4dbdc8952ea40b910eb30cb7685c2c865f7fe37c4ec705c568"
Nov 02 12:50:35.057682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:35.057642    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" event={"ID":"2844d7db-1a7a-4ebd-93c8-c49944bbb954","Type":"ContainerDied","Data":"493c25b10bb68b4dbdc8952ea40b910eb30cb7685c2c865f7fe37c4ec705c568"}
Nov 02 12:50:35.057813 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:35.057719    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl" event={"ID":"2844d7db-1a7a-4ebd-93c8-c49944bbb954","Type":"ContainerDied","Data":"ca88e7fbff64d0b516a5e4bc0a732659016a92557cfe8087c8c66a7071f33a20"}
Nov 02 12:50:35.057813 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:35.057721    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"
Nov 02 12:50:35.172215 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:35.172169    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"]
Nov 02 12:50:35.179706 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:35.179667    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-7pqzl"]
Nov 02 12:50:35.804197 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:35.804163    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:36.017618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:36.017583    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" path="/var/lib/kubelet/pods/2844d7db-1a7a-4ebd-93c8-c49944bbb954/volumes"
Nov 02 12:50:36.455858 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:36.455831    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:37.947659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:37.947562    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:37.997285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:37.997259    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:50:38.077617 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.077583    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-data\") pod \"11648a63-a758-4f5e-be54-976d4e234612\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") "
Nov 02 12:50:38.077838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.077637    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-socket\") pod \"11648a63-a758-4f5e-be54-976d4e234612\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") "
Nov 02 12:50:38.077838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.077699    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-istio-token\" (UniqueName: \"kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-istio-token\") pod \"11648a63-a758-4f5e-be54-976d4e234612\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") "
Nov 02 12:50:38.077838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.077756    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-credential-socket\") pod \"11648a63-a758-4f5e-be54-976d4e234612\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") "
Nov 02 12:50:38.077838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.077791    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11648a63-a758-4f5e-be54-976d4e234612-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11648a63-a758-4f5e-be54-976d4e234612-istio-podinfo\") pod \"11648a63-a758-4f5e-be54-976d4e234612\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") "
Nov 02 12:50:38.078118 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.077892    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11648a63-a758-4f5e-be54-976d4e234612-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11648a63-a758-4f5e-be54-976d4e234612-istiod-ca-cert\") pod \"11648a63-a758-4f5e-be54-976d4e234612\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") "
Nov 02 12:50:38.078118 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.077948    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-socket" pod "11648a63-a758-4f5e-be54-976d4e234612" (UID: "11648a63-a758-4f5e-be54-976d4e234612"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:38.078118 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.078062    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-kube-api-access-knn7v\" (UniqueName: \"kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-kube-api-access-knn7v\") pod \"11648a63-a758-4f5e-be54-976d4e234612\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") "
Nov 02 12:50:38.078118 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.078116    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-envoy\") pod \"11648a63-a758-4f5e-be54-976d4e234612\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") "
Nov 02 12:50:38.078327 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.078186    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-credential-socket" pod "11648a63-a758-4f5e-be54-976d4e234612" (UID: "11648a63-a758-4f5e-be54-976d4e234612"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:38.079388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.078467    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-certs\") pod \"11648a63-a758-4f5e-be54-976d4e234612\" (UID: \"11648a63-a758-4f5e-be54-976d4e234612\") "
Nov 02 12:50:38.079388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.078510    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11648a63-a758-4f5e-be54-976d4e234612-istiod-ca-cert" pod "11648a63-a758-4f5e-be54-976d4e234612" (UID: "11648a63-a758-4f5e-be54-976d4e234612"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:50:38.079388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.079065    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-certs" pod "11648a63-a758-4f5e-be54-976d4e234612" (UID: "11648a63-a758-4f5e-be54-976d4e234612"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:38.079388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.079290    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:38.079388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.079311    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:38.079388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.079327    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:38.079388 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.079342    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11648a63-a758-4f5e-be54-976d4e234612-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:38.079970 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.079928    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-data" pod "11648a63-a758-4f5e-be54-976d4e234612" (UID: "11648a63-a758-4f5e-be54-976d4e234612"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:38.080700 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.080678    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-istio-token" pod "11648a63-a758-4f5e-be54-976d4e234612" (UID: "11648a63-a758-4f5e-be54-976d4e234612"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:38.080851 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.080770    5879 generic.go:356] "Generic (PLEG): container finished" podID="11648a63-a758-4f5e-be54-976d4e234612" containerID="67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750" exitCode=0
Nov 02 12:50:38.080938 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.080847    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"
Nov 02 12:50:38.082800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.081044    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" event={"ID":"11648a63-a758-4f5e-be54-976d4e234612","Type":"ContainerDied","Data":"67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750"}
Nov 02 12:50:38.082800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.081099    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb" event={"ID":"11648a63-a758-4f5e-be54-976d4e234612","Type":"ContainerDied","Data":"1da9355081f2307cf58f52f23d1ce293b1331dcaa1102bc3df70971baeadd501"}
Nov 02 12:50:38.082800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.081130    5879 scope.go:122] "RemoveContainer" containerID="67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750"
Nov 02 12:50:38.085671 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.085364    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11648a63-a758-4f5e-be54-976d4e234612-istio-podinfo" pod "11648a63-a758-4f5e-be54-976d4e234612" (UID: "11648a63-a758-4f5e-be54-976d4e234612"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:50:38.087387 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.087360    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-kube-api-access-knn7v" pod "11648a63-a758-4f5e-be54-976d4e234612" (UID: "11648a63-a758-4f5e-be54-976d4e234612"). InnerVolumeSpecName "kube-api-access-knn7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:50:38.087554 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.087529    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-envoy" pod "11648a63-a758-4f5e-be54-976d4e234612" (UID: "11648a63-a758-4f5e-be54-976d4e234612"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:50:38.095375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.095351    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:38.113985 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.113959    5879 scope.go:122] "RemoveContainer" containerID="67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750"
Nov 02 12:50:38.114485 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:50:38.114460    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750\": container with ID starting with 67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750 not found: ID does not exist" containerID="67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750"
Nov 02 12:50:38.114569 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.114497    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750"} err="failed to get container status \"67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750\": rpc error: code = NotFound desc = could not find container \"67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750\": container with ID starting with 67a88b31ce094e4ea29666f3be83cbd2cf515d67c44f65f3ff8d023e5aa13750 not found: ID does not exist"
Nov 02 12:50:38.181421 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.181369    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:38.181421 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.181410    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11648a63-a758-4f5e-be54-976d4e234612-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:38.181421 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.181424    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knn7v\" (UniqueName: \"kubernetes.io/projected/11648a63-a758-4f5e-be54-976d4e234612-kube-api-access-knn7v\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:38.181763 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.181438    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:38.181763 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.181450    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11648a63-a758-4f5e-be54-976d4e234612-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:50:38.228029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.227913    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:38.431288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.431257    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"]
Nov 02 12:50:38.445217 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:38.445188    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-k8ncb"]
Nov 02 12:50:39.023798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:39.023763    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:39.577567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:39.577533    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:40.018723 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:40.018681    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11648a63-a758-4f5e-be54-976d4e234612" path="/var/lib/kubelet/pods/11648a63-a758-4f5e-be54-976d4e234612/volumes"
Nov 02 12:50:40.435284 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:40.435247    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:42.574590 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:42.574554    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:43.277434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.277405    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"]
Nov 02 12:50:43.277969 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.277956    5879 cpu_manager.go:414] "containerMap: removing container" podUID="11648a63-a758-4f5e-be54-976d4e234612" containerName="istio-proxy"
Nov 02 12:50:43.278046 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.277971    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11648a63-a758-4f5e-be54-976d4e234612" containerName="istio-proxy"
Nov 02 12:50:43.278046 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.278016    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy"
Nov 02 12:50:43.278046 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.278024    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy"
Nov 02 12:50:43.278153 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.278133    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="11648a63-a758-4f5e-be54-976d4e234612" containerName="istio-proxy"
Nov 02 12:50:43.278153 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.278144    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2844d7db-1a7a-4ebd-93c8-c49944bbb954" containerName="istio-proxy"
Nov 02 12:50:43.283079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.283062    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.308522 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.308494    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"]
Nov 02 12:50:43.358321 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.358286    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.358489 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.358342    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.358489 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.358461    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.358630 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.358576    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.358684 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.358648    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.358741 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.358719    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.358871 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.358849    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.359025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.358914    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxpcc\" (UniqueName: \"kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-kube-api-access-lxpcc\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.359025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.358946    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.384756 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.384733    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-9750c7-openshift-default-dockercfg-6f96j\""
Nov 02 12:50:43.442486 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.442459    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:43.460367 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.460341    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.460476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.460396    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxpcc\" (UniqueName: \"kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-kube-api-access-lxpcc\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.460476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.460450    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.460567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.460492    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.460567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.460539    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.460666 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.460590    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.460721 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.460706    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.460801 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.460780    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.460855 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.460818    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.460905 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.460862    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.461385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.461324    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.461385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.461351    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.461602 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.461579    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.461795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.461770    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.463400 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.463378    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.489750 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.489686    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.490538 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.490478    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxpcc\" (UniqueName: \"kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-kube-api-access-lxpcc\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.491076 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.491051    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.603592 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.603515    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:43.924375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.924333    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"]
Nov 02 12:50:43.941426 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:50:43.941396    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0c26de_cbe1_462a_9d0c_ada687f84a78.slice/crio-b9c9ca06ac6a819c268c306e2a3184aa7a14aec4658dadfaa8c8c97ab16bd6ca WatchSource:0}: Error finding container b9c9ca06ac6a819c268c306e2a3184aa7a14aec4658dadfaa8c8c97ab16bd6ca: Status 404 returned error can't find the container with id b9c9ca06ac6a819c268c306e2a3184aa7a14aec4658dadfaa8c8c97ab16bd6ca
Nov 02 12:50:43.944044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.943959    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:50:43.944128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.944046    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:50:43.944128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:43.944079    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:50:44.054251 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:44.054221    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"
Nov 02 12:50:44.132100 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:44.132064    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv" event={"ID":"ec0c26de-cbe1-462a-9d0c-ada687f84a78","Type":"ContainerStarted","Data":"8f145e506bd40b11daa786109cb081e838c7563fe81e0f12ddb213eab90dc6ac"}
Nov 02 12:50:44.132270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:44.132131    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv" event={"ID":"ec0c26de-cbe1-462a-9d0c-ada687f84a78","Type":"ContainerStarted","Data":"b9c9ca06ac6a819c268c306e2a3184aa7a14aec4658dadfaa8c8c97ab16bd6ca"}
Nov 02 12:50:44.179769 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:44.179439    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv" podStartSLOduration=1.179421451 podStartE2EDuration="1.179421451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:44.178701814 +0000 UTC m=+1390.285404664" watchObservedRunningTime="2026-11-02 12:50:44.179421451 +0000 UTC m=+1390.286124301"
Nov 02 12:50:44.445927 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:44.445821    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:44.606126 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:44.606085    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:44.834600 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:50:44.834509    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:50:45.362575 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:45.362538    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:45.503018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:45.493950    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:45.608923 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:45.608883    5879 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 02 12:50:45.609427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:45.608946    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv" podUID="ec0c26de-cbe1-462a-9d0c-ada687f84a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 02 12:50:45.663541 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:45.663510    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:46.150476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:46.150438    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:46.309339 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:46.309297    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:46.517452 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:46.517348    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:46.614016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:46.608388    5879 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 02 12:50:46.614016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:46.608446    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv" podUID="ec0c26de-cbe1-462a-9d0c-ada687f84a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 02 12:50:47.608140 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:47.608101    5879 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 02 12:50:47.608447 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:47.608414    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv" podUID="ec0c26de-cbe1-462a-9d0c-ada687f84a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 02 12:50:48.173707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:48.173676    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:48.606158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:48.605317    5879 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 02 12:50:48.606158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:48.605386    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv" podUID="ec0c26de-cbe1-462a-9d0c-ada687f84a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 02 12:50:49.276956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:49.276923    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:49.604427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:49.604340    5879 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body=
Nov 02 12:50:49.605058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:49.604665    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv" podUID="ec0c26de-cbe1-462a-9d0c-ada687f84a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused"
Nov 02 12:50:49.767567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:49.767533    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:49.915228 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:49.915195    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:50.049128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:50.049088    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:50.609080 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:50.609051    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:50.609556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:50.609529    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:50.610288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:50.610268    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:50:51.652219 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:51.652184    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:52.777320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:52.777281    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:52.912872 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:52.912834    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:53.048406 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:53.048327    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:53.183690 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:53.183656    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:53.890125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:53.890089    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:54.079384 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:54.079353    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:54.218538 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:54.218439    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:55.567535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:55.567486    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:55.698109 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:55.698072    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:56.331552 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.331519    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:56.704963 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.704931    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:56.751110 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.751079    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"]
Nov 02 12:50:56.755119 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.755101    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:50:56.772246 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.772224    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:50:56.775864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.775840    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"]
Nov 02 12:50:56.785866 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.785845    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:50:56.848287 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.848246    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43636316-twlzl\" (UID: \"7965e746-da50-44ab-aa60-67505ec31da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:50:56.848473 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.848446    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43636316-twlzl\" (UID: \"7965e746-da50-44ab-aa60-67505ec31da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:50:56.848643 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.848620    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb9gc\" (UniqueName: \"kubernetes.io/projected/7965e746-da50-44ab-aa60-67505ec31da8-kube-api-access-gb9gc\") pod \"maas-api-revoke-sub-43636316-twlzl\" (UID: \"7965e746-da50-44ab-aa60-67505ec31da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:50:56.950250 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.950217    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43636316-twlzl\" (UID: \"7965e746-da50-44ab-aa60-67505ec31da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:50:56.950431 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.950384    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43636316-twlzl\" (UID: \"7965e746-da50-44ab-aa60-67505ec31da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:50:56.950573 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.950552    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb9gc\" (UniqueName: \"kubernetes.io/projected/7965e746-da50-44ab-aa60-67505ec31da8-kube-api-access-gb9gc\") pod \"maas-api-revoke-sub-43636316-twlzl\" (UID: \"7965e746-da50-44ab-aa60-67505ec31da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:50:56.950979 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.950958    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43636316-twlzl\" (UID: \"7965e746-da50-44ab-aa60-67505ec31da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:50:56.953204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.953181    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43636316-twlzl\" (UID: \"7965e746-da50-44ab-aa60-67505ec31da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:50:56.975764 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:56.975695    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb9gc\" (UniqueName: \"kubernetes.io/projected/7965e746-da50-44ab-aa60-67505ec31da8-kube-api-access-gb9gc\") pod \"maas-api-revoke-sub-43636316-twlzl\" (UID: \"7965e746-da50-44ab-aa60-67505ec31da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:50:57.069525 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:57.069488    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:50:57.371167 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:50:57.371132    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7965e746_da50_44ab_aa60_67505ec31da8.slice/crio-f454d0d55873afd5b5bca7e9152ca263f4f84afc5aa7fec390f55b5905cedc8b WatchSource:0}: Error finding container f454d0d55873afd5b5bca7e9152ca263f4f84afc5aa7fec390f55b5905cedc8b: Status 404 returned error can't find the container with id f454d0d55873afd5b5bca7e9152ca263f4f84afc5aa7fec390f55b5905cedc8b
Nov 02 12:50:57.371330 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:57.371289    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"]
Nov 02 12:50:57.965755 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:57.965675    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:58.238514 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:58.238433    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl" event={"ID":"7965e746-da50-44ab-aa60-67505ec31da8","Type":"ContainerStarted","Data":"044d2dfa9cac178033e26cf6af9949be5167da46bd9ca01cbd96aa7c21eff9a6"}
Nov 02 12:50:58.238514 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:58.238483    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl" event={"ID":"7965e746-da50-44ab-aa60-67505ec31da8","Type":"ContainerStarted","Data":"f454d0d55873afd5b5bca7e9152ca263f4f84afc5aa7fec390f55b5905cedc8b"}
Nov 02 12:50:58.284158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:58.284112    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl" podStartSLOduration=2.284096433 podStartE2EDuration="2.284096433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:50:58.281160275 +0000 UTC m=+1404.387863127" watchObservedRunningTime="2026-11-02 12:50:58.284096433 +0000 UTC m=+1404.390799344"
Nov 02 12:50:58.498257 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:58.498167    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:58.624716 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:58.624683    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:50:59.245944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:59.245909    5879 generic.go:356] "Generic (PLEG): container finished" podID="7965e746-da50-44ab-aa60-67505ec31da8" containerID="044d2dfa9cac178033e26cf6af9949be5167da46bd9ca01cbd96aa7c21eff9a6" exitCode=0
Nov 02 12:50:59.246351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:59.246026    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl" event={"ID":"7965e746-da50-44ab-aa60-67505ec31da8","Type":"ContainerDied","Data":"044d2dfa9cac178033e26cf6af9949be5167da46bd9ca01cbd96aa7c21eff9a6"}
Nov 02 12:50:59.380595 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:50:59.380564    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:00.153726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:00.153687    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:00.511304 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:00.511280    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:51:00.699436 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:00.699404    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7965e746-da50-44ab-aa60-67505ec31da8-kube-api-access-gb9gc\" (UniqueName: \"kubernetes.io/projected/7965e746-da50-44ab-aa60-67505ec31da8-kube-api-access-gb9gc\") pod \"7965e746-da50-44ab-aa60-67505ec31da8\" (UID: \"7965e746-da50-44ab-aa60-67505ec31da8\") "
Nov 02 12:51:00.699630 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:00.699513    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-xks-ca\") pod \"7965e746-da50-44ab-aa60-67505ec31da8\" (UID: \"7965e746-da50-44ab-aa60-67505ec31da8\") "
Nov 02 12:51:00.699692 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:00.699662    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-openshift-ca\") pod \"7965e746-da50-44ab-aa60-67505ec31da8\" (UID: \"7965e746-da50-44ab-aa60-67505ec31da8\") "
Nov 02 12:51:00.700070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:00.700039    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-openshift-ca" pod "7965e746-da50-44ab-aa60-67505ec31da8" (UID: "7965e746-da50-44ab-aa60-67505ec31da8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:00.700291 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:00.700273    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:00.701672 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:00.701646    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7965e746-da50-44ab-aa60-67505ec31da8-kube-api-access-gb9gc" pod "7965e746-da50-44ab-aa60-67505ec31da8" (UID: "7965e746-da50-44ab-aa60-67505ec31da8"). InnerVolumeSpecName "kube-api-access-gb9gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:00.701760 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:00.701700    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-xks-ca" pod "7965e746-da50-44ab-aa60-67505ec31da8" (UID: "7965e746-da50-44ab-aa60-67505ec31da8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:00.802186 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:00.802101    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7965e746-da50-44ab-aa60-67505ec31da8-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:00.802186 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:00.802130    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb9gc\" (UniqueName: \"kubernetes.io/projected/7965e746-da50-44ab-aa60-67505ec31da8-kube-api-access-gb9gc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:01.263607 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:01.263567    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl" event={"ID":"7965e746-da50-44ab-aa60-67505ec31da8","Type":"ContainerDied","Data":"f454d0d55873afd5b5bca7e9152ca263f4f84afc5aa7fec390f55b5905cedc8b"}
Nov 02 12:51:01.264072 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:01.264047    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f454d0d55873afd5b5bca7e9152ca263f4f84afc5aa7fec390f55b5905cedc8b"
Nov 02 12:51:01.264072 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:01.263941    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"
Nov 02 12:51:01.827646 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:01.827614    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:02.798119 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:02.798087    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:02.937603 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:02.937574    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:03.148458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:03.148417    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:03.651398 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:03.651341    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:05.776059 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:05.776021    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:07.281386 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:07.281340    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:07.967058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:07.967027    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:09.637427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:09.637375    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:09.776006 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:09.775938    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:10.121591 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:10.121563    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:10.343975 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:10.343943    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:10.729436 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:10.729389    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:11.342580 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:11.342544    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:12.660410 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:12.660378    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:13.202717 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.202677    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:13.587181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.587092    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"]
Nov 02 12:51:13.587729 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.587714    5879 cpu_manager.go:414] "containerMap: removing container" podUID="7965e746-da50-44ab-aa60-67505ec31da8" containerName="revoke-keys"
Nov 02 12:51:13.587796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.587730    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7965e746-da50-44ab-aa60-67505ec31da8" containerName="revoke-keys"
Nov 02 12:51:13.587858 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.587852    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="7965e746-da50-44ab-aa60-67505ec31da8" containerName="revoke-keys"
Nov 02 12:51:13.592696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.592675    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:13.612319 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.612285    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"]
Nov 02 12:51:13.621876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.621846    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:51:13.622263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.622240    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:51:13.740342 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.740306    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txzcb\" (UniqueName: \"kubernetes.io/projected/3500de69-4911-4539-a6a9-48830aa2a6bf-kube-api-access-txzcb\") pod \"maas-api-revoke-sub-b9cb73b3-69qgl\" (UID: \"3500de69-4911-4539-a6a9-48830aa2a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:13.740781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.740510    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9cb73b3-69qgl\" (UID: \"3500de69-4911-4539-a6a9-48830aa2a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:13.740781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.740533    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9cb73b3-69qgl\" (UID: \"3500de69-4911-4539-a6a9-48830aa2a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:13.842444 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.842364    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txzcb\" (UniqueName: \"kubernetes.io/projected/3500de69-4911-4539-a6a9-48830aa2a6bf-kube-api-access-txzcb\") pod \"maas-api-revoke-sub-b9cb73b3-69qgl\" (UID: \"3500de69-4911-4539-a6a9-48830aa2a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:13.842609 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.842515    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9cb73b3-69qgl\" (UID: \"3500de69-4911-4539-a6a9-48830aa2a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:13.842609 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.842536    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9cb73b3-69qgl\" (UID: \"3500de69-4911-4539-a6a9-48830aa2a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:13.843171 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.843152    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9cb73b3-69qgl\" (UID: \"3500de69-4911-4539-a6a9-48830aa2a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:13.845070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.845040    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9cb73b3-69qgl\" (UID: \"3500de69-4911-4539-a6a9-48830aa2a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:13.867755 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.867722    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txzcb\" (UniqueName: \"kubernetes.io/projected/3500de69-4911-4539-a6a9-48830aa2a6bf-kube-api-access-txzcb\") pod \"maas-api-revoke-sub-b9cb73b3-69qgl\" (UID: \"3500de69-4911-4539-a6a9-48830aa2a6bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:13.906962 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:13.906933    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:14.213510 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:51:14.213476    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3500de69_4911_4539_a6a9_48830aa2a6bf.slice/crio-8a29dd6f24cb78bced5614a9854bf9538d57b5eb5d3912e41a0ace125b8de189 WatchSource:0}: Error finding container 8a29dd6f24cb78bced5614a9854bf9538d57b5eb5d3912e41a0ace125b8de189: Status 404 returned error can't find the container with id 8a29dd6f24cb78bced5614a9854bf9538d57b5eb5d3912e41a0ace125b8de189
Nov 02 12:51:14.214669 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:14.214647    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"]
Nov 02 12:51:14.469373 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:14.469290    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl" event={"ID":"3500de69-4911-4539-a6a9-48830aa2a6bf","Type":"ContainerStarted","Data":"8a29dd6f24cb78bced5614a9854bf9538d57b5eb5d3912e41a0ace125b8de189"}
Nov 02 12:51:14.821118 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:14.821032    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:14.822497 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:51:14.822469    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:51:14.958185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:14.958152    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:15.069230 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:15.069196    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"]
Nov 02 12:51:15.121317 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:15.121285    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:15.272898 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:15.272862    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:15.479201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:15.479155    5879 generic.go:356] "Generic (PLEG): container finished" podID="3500de69-4911-4539-a6a9-48830aa2a6bf" containerID="cc385bc40750e18556acd15c7c7b01f372312899fa60f478fe3b037acf00b08b" exitCode=0
Nov 02 12:51:15.479412 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:15.479241    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl" event={"ID":"3500de69-4911-4539-a6a9-48830aa2a6bf","Type":"ContainerDied","Data":"cc385bc40750e18556acd15c7c7b01f372312899fa60f478fe3b037acf00b08b"}
Nov 02 12:51:16.664428 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:16.664403    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:16.783283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:16.783190    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3500de69-4911-4539-a6a9-48830aa2a6bf-kube-api-access-txzcb\" (UniqueName: \"kubernetes.io/projected/3500de69-4911-4539-a6a9-48830aa2a6bf-kube-api-access-txzcb\") pod \"3500de69-4911-4539-a6a9-48830aa2a6bf\" (UID: \"3500de69-4911-4539-a6a9-48830aa2a6bf\") "
Nov 02 12:51:16.783283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:16.783278    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-xks-ca\") pod \"3500de69-4911-4539-a6a9-48830aa2a6bf\" (UID: \"3500de69-4911-4539-a6a9-48830aa2a6bf\") "
Nov 02 12:51:16.783521 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:16.783365    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-openshift-ca\") pod \"3500de69-4911-4539-a6a9-48830aa2a6bf\" (UID: \"3500de69-4911-4539-a6a9-48830aa2a6bf\") "
Nov 02 12:51:16.783770 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:16.783747    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-openshift-ca" pod "3500de69-4911-4539-a6a9-48830aa2a6bf" (UID: "3500de69-4911-4539-a6a9-48830aa2a6bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:16.784035 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:16.783985    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:16.785537 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:16.785510    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-xks-ca" pod "3500de69-4911-4539-a6a9-48830aa2a6bf" (UID: "3500de69-4911-4539-a6a9-48830aa2a6bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:16.785630 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:16.785565    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3500de69-4911-4539-a6a9-48830aa2a6bf-kube-api-access-txzcb" pod "3500de69-4911-4539-a6a9-48830aa2a6bf" (UID: "3500de69-4911-4539-a6a9-48830aa2a6bf"). InnerVolumeSpecName "kube-api-access-txzcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:16.885284 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:16.885253    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3500de69-4911-4539-a6a9-48830aa2a6bf-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:16.885284 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:16.885281    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txzcb\" (UniqueName: \"kubernetes.io/projected/3500de69-4911-4539-a6a9-48830aa2a6bf-kube-api-access-txzcb\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:16.976041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:16.975968    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:17.494577 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:17.494539    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl" event={"ID":"3500de69-4911-4539-a6a9-48830aa2a6bf","Type":"ContainerDied","Data":"8a29dd6f24cb78bced5614a9854bf9538d57b5eb5d3912e41a0ace125b8de189"}
Nov 02 12:51:17.494577 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:17.494576    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"
Nov 02 12:51:17.494803 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:17.494598    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a29dd6f24cb78bced5614a9854bf9538d57b5eb5d3912e41a0ace125b8de189"
Nov 02 12:51:18.443040 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:18.442973    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"]
Nov 02 12:51:18.582030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:18.581952    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:18.703288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:18.703183    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:18.905348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:18.905308    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:19.168718 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:19.168685    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:20.840918 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:20.840880    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:22.856267 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:22.856227    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:23.443678 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:23.443639    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx" podUID="8dfb5287-27fe-483b-a7c3-45e6fe896c95" containerName="payload-pre-processing" containerID="cri-o://8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c" gracePeriod=25
Nov 02 12:51:23.727261 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:23.727233    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"
Nov 02 12:51:23.797443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:23.797411    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dfb5287-27fe-483b-a7c3-45e6fe896c95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfb5287-27fe-483b-a7c3-45e6fe896c95-plugins-config-volume\") pod \"8dfb5287-27fe-483b-a7c3-45e6fe896c95\" (UID: \"8dfb5287-27fe-483b-a7c3-45e6fe896c95\") "
Nov 02 12:51:23.797859 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:23.797831    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dfb5287-27fe-483b-a7c3-45e6fe896c95-plugins-config-volume" pod "8dfb5287-27fe-483b-a7c3-45e6fe896c95" (UID: "8dfb5287-27fe-483b-a7c3-45e6fe896c95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:23.798019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:23.797930    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dfb5287-27fe-483b-a7c3-45e6fe896c95-kube-api-access-p88vm\" (UniqueName: \"kubernetes.io/projected/8dfb5287-27fe-483b-a7c3-45e6fe896c95-kube-api-access-p88vm\") pod \"8dfb5287-27fe-483b-a7c3-45e6fe896c95\" (UID: \"8dfb5287-27fe-483b-a7c3-45e6fe896c95\") "
Nov 02 12:51:23.798584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:23.798558    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dfb5287-27fe-483b-a7c3-45e6fe896c95-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:23.800438 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:23.800408    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfb5287-27fe-483b-a7c3-45e6fe896c95-kube-api-access-p88vm" pod "8dfb5287-27fe-483b-a7c3-45e6fe896c95" (UID: "8dfb5287-27fe-483b-a7c3-45e6fe896c95"). InnerVolumeSpecName "kube-api-access-p88vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:23.900892 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:23.900855    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p88vm\" (UniqueName: \"kubernetes.io/projected/8dfb5287-27fe-483b-a7c3-45e6fe896c95-kube-api-access-p88vm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:24.233461 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.233420    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:24.383635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.383601    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:24.517821 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.517780    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:24.541576 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.541544    5879 generic.go:356] "Generic (PLEG): container finished" podID="8dfb5287-27fe-483b-a7c3-45e6fe896c95" containerID="8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c" exitCode=0
Nov 02 12:51:24.541807 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.541623    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"
Nov 02 12:51:24.541807 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.541628    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx" event={"ID":"8dfb5287-27fe-483b-a7c3-45e6fe896c95","Type":"ContainerDied","Data":"8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c"}
Nov 02 12:51:24.541807 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.541681    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx" event={"ID":"8dfb5287-27fe-483b-a7c3-45e6fe896c95","Type":"ContainerDied","Data":"510455f30a0945b2fc2ea2cefc087319ca6fb74e6765bb22d00188f30e5af4fa"}
Nov 02 12:51:24.541807 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.541705    5879 scope.go:122] "RemoveContainer" containerID="8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c"
Nov 02 12:51:24.552302 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.552283    5879 scope.go:122] "RemoveContainer" containerID="8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c"
Nov 02 12:51:24.552627 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:51:24.552605    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c\": container with ID starting with 8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c not found: ID does not exist" containerID="8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c"
Nov 02 12:51:24.552671 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.552638    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c"} err="failed to get container status \"8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c\": rpc error: code = NotFound desc = could not find container \"8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c\": container with ID starting with 8f9cad27cd6bb1004c6ac3ac8c737377d0a61c345f835718f5d97f16b96ef65c not found: ID does not exist"
Nov 02 12:51:24.597974 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.597940    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"]
Nov 02 12:51:24.611370 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.611338    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-d3a281-58f7476895-tpwjx"]
Nov 02 12:51:24.898225 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:24.898195    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:25.301188 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:25.301094    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:25.835199 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:25.835161    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:26.019894 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:26.019866    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dfb5287-27fe-483b-a7c3-45e6fe896c95" path="/var/lib/kubelet/pods/8dfb5287-27fe-483b-a7c3-45e6fe896c95/volumes"
Nov 02 12:51:26.046440 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:26.046412    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:26.328506 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:26.328475    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:28.428151 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.428083    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"]
Nov 02 12:51:28.433123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.429872    5879 cpu_manager.go:414] "containerMap: removing container" podUID="8dfb5287-27fe-483b-a7c3-45e6fe896c95" containerName="payload-pre-processing"
Nov 02 12:51:28.433123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.429895    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dfb5287-27fe-483b-a7c3-45e6fe896c95" containerName="payload-pre-processing"
Nov 02 12:51:28.433123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.429920    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3500de69-4911-4539-a6a9-48830aa2a6bf" containerName="revoke-keys"
Nov 02 12:51:28.433123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.429929    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3500de69-4911-4539-a6a9-48830aa2a6bf" containerName="revoke-keys"
Nov 02 12:51:28.433123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.431101    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dfb5287-27fe-483b-a7c3-45e6fe896c95" containerName="payload-pre-processing"
Nov 02 12:51:28.433123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.431122    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3500de69-4911-4539-a6a9-48830aa2a6bf" containerName="revoke-keys"
Nov 02 12:51:28.435085 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.435064    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.464607 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.464574    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"]
Nov 02 12:51:28.518036 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.513032    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.518350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.518321    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7036741f-e614-4dc6-b368-10e452e8310a-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.518531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.518514    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.518884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.518865    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.519144 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.519122    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.519368 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.519328    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.519512 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.519496    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7036741f-e614-4dc6-b368-10e452e8310a-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.519638 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.519622    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.519801 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.519784    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkrx\" (UniqueName: \"kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-kube-api-access-zpkrx\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.608740 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.608608    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:28.627018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.622018    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.627382    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.633197    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.633257    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7036741f-e614-4dc6-b368-10e452e8310a-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.633288    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.633372    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkrx\" (UniqueName: \"kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-kube-api-access-zpkrx\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.633470    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-certs\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.633503    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.633539    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7036741f-e614-4dc6-b368-10e452e8310a-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.633615    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.634083    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-credential-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.634323    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-data\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.634949    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-socket\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.638030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.634970    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7036741f-e614-4dc6-b368-10e452e8310a-istiod-ca-cert\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.684525 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.684414    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7036741f-e614-4dc6-b368-10e452e8310a-istio-podinfo\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.684937 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.684893    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-envoy\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.696299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.696236    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkrx\" (UniqueName: \"kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-kube-api-access-zpkrx\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.696923 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.696873    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-istio-token\") pod \"e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") " pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:28.765865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:28.765794    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:29.125127 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.125104    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"]
Nov 02 12:51:29.127101 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.127068    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:51:29.127191 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.127129    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:51:29.127191 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.127159    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:51:29.579849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.579753    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222" event={"ID":"7036741f-e614-4dc6-b368-10e452e8310a","Type":"ContainerStarted","Data":"ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659"}
Nov 02 12:51:29.579849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.579815    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222" event={"ID":"7036741f-e614-4dc6-b368-10e452e8310a","Type":"ContainerStarted","Data":"2d1c274dd4529e53a16d251ccfc197c15bcb8e1bc21ca413660c4eb1957254c7"}
Nov 02 12:51:29.588111 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.588081    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"]
Nov 02 12:51:29.592405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.592375    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:29.608946 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.608903    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:51:29.609146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.609106    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:51:29.613118 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.613089    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"]
Nov 02 12:51:29.627337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.627281    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222" podStartSLOduration=1.6272650560000002 podStartE2EDuration="1.627265056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:29.626780411 +0000 UTC m=+1435.733483263" watchObservedRunningTime="2026-11-02 12:51:29.627265056 +0000 UTC m=+1435.733967908"
Nov 02 12:51:29.755843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.755806    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f9a5c5-c6f2n\" (UID: \"912751e5-8e00-41f4-b45e-6a8244add550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:29.756113 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.756090    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fllx6\" (UniqueName: \"kubernetes.io/projected/912751e5-8e00-41f4-b45e-6a8244add550-kube-api-access-fllx6\") pod \"maas-api-revoke-sub-41f9a5c5-c6f2n\" (UID: \"912751e5-8e00-41f4-b45e-6a8244add550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:29.756223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.756210    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f9a5c5-c6f2n\" (UID: \"912751e5-8e00-41f4-b45e-6a8244add550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:29.766450 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.766429    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:29.857203 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.857113    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f9a5c5-c6f2n\" (UID: \"912751e5-8e00-41f4-b45e-6a8244add550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:29.857400 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.857251    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f9a5c5-c6f2n\" (UID: \"912751e5-8e00-41f4-b45e-6a8244add550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:29.857466 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.857408    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fllx6\" (UniqueName: \"kubernetes.io/projected/912751e5-8e00-41f4-b45e-6a8244add550-kube-api-access-fllx6\") pod \"maas-api-revoke-sub-41f9a5c5-c6f2n\" (UID: \"912751e5-8e00-41f4-b45e-6a8244add550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:29.857845 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.857827    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f9a5c5-c6f2n\" (UID: \"912751e5-8e00-41f4-b45e-6a8244add550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:29.859788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.859765    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f9a5c5-c6f2n\" (UID: \"912751e5-8e00-41f4-b45e-6a8244add550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:29.882354 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.882326    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fllx6\" (UniqueName: \"kubernetes.io/projected/912751e5-8e00-41f4-b45e-6a8244add550-kube-api-access-fllx6\") pod \"maas-api-revoke-sub-41f9a5c5-c6f2n\" (UID: \"912751e5-8e00-41f4-b45e-6a8244add550\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:29.908915 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:29.908883    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:30.091785 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.091743    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s" podUID="61768aba-42e4-4b0c-87e3-53c4e4e17c66" containerName="main" containerID="cri-o://314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037" gracePeriod=45
Nov 02 12:51:30.249635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.249611    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"]
Nov 02 12:51:30.250205 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:51:30.250181    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912751e5_8e00_41f4_b45e_6a8244add550.slice/crio-650018e114590f30a30865fb87f45a280af6ec86ab122bf40068a739715a1938 WatchSource:0}: Error finding container 650018e114590f30a30865fb87f45a280af6ec86ab122bf40068a739715a1938: Status 404 returned error can't find the container with id 650018e114590f30a30865fb87f45a280af6ec86ab122bf40068a739715a1938
Nov 02 12:51:30.274903 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.274870    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:30.434567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.434541    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:51:30.568337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.568310    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61768aba-42e4-4b0c-87e3-53c4e4e17c66-kube-api-access-nbjsp\" (UniqueName: \"kubernetes.io/projected/61768aba-42e4-4b0c-87e3-53c4e4e17c66-kube-api-access-nbjsp\") pod \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") "
Nov 02 12:51:30.568530 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.568413    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-home\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-home\") pod \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") "
Nov 02 12:51:30.568530 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.568458    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-dshm\") pod \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") "
Nov 02 12:51:30.568530 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.568485    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tmp-dir\") pod \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") "
Nov 02 12:51:30.568731 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.568620    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-model-cache\") pod \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") "
Nov 02 12:51:30.568731 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.568705    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-home" pod "61768aba-42e4-4b0c-87e3-53c4e4e17c66" (UID: "61768aba-42e4-4b0c-87e3-53c4e4e17c66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:30.568829 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.568751    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tls-certs\" (UniqueName: \"kubernetes.io/secret/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tls-certs\") pod \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\" (UID: \"61768aba-42e4-4b0c-87e3-53c4e4e17c66\") "
Nov 02 12:51:30.568958 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.568932    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-model-cache" pod "61768aba-42e4-4b0c-87e3-53c4e4e17c66" (UID: "61768aba-42e4-4b0c-87e3-53c4e4e17c66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:30.569453 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.569422    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.569514 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.569451    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.569594 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.569578    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tmp-dir" pod "61768aba-42e4-4b0c-87e3-53c4e4e17c66" (UID: "61768aba-42e4-4b0c-87e3-53c4e4e17c66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:30.571277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.571252    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61768aba-42e4-4b0c-87e3-53c4e4e17c66-kube-api-access-nbjsp" pod "61768aba-42e4-4b0c-87e3-53c4e4e17c66" (UID: "61768aba-42e4-4b0c-87e3-53c4e4e17c66"). InnerVolumeSpecName "kube-api-access-nbjsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:30.571277 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.571269    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tls-certs" pod "61768aba-42e4-4b0c-87e3-53c4e4e17c66" (UID: "61768aba-42e4-4b0c-87e3-53c4e4e17c66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:30.571552 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.571535    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-dshm" pod "61768aba-42e4-4b0c-87e3-53c4e4e17c66" (UID: "61768aba-42e4-4b0c-87e3-53c4e4e17c66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:30.618348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.618314    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n" event={"ID":"912751e5-8e00-41f4-b45e-6a8244add550","Type":"ContainerStarted","Data":"650018e114590f30a30865fb87f45a280af6ec86ab122bf40068a739715a1938"}
Nov 02 12:51:30.619793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.619768    5879 generic.go:356] "Generic (PLEG): container finished" podID="61768aba-42e4-4b0c-87e3-53c4e4e17c66" containerID="314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037" exitCode=0
Nov 02 12:51:30.619937 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.619821    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"
Nov 02 12:51:30.619937 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.619849    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s" event={"ID":"61768aba-42e4-4b0c-87e3-53c4e4e17c66","Type":"ContainerDied","Data":"314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037"}
Nov 02 12:51:30.619937 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.619887    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s" event={"ID":"61768aba-42e4-4b0c-87e3-53c4e4e17c66","Type":"ContainerDied","Data":"01b69f5a63bee990358ca426a127df5ca6e1a4b30cd6e746c68534ffd4e0c58c"}
Nov 02 12:51:30.619937 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.619904    5879 scope.go:122] "RemoveContainer" containerID="314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037"
Nov 02 12:51:30.634196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.634172    5879 scope.go:122] "RemoveContainer" containerID="314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037"
Nov 02 12:51:30.634487 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:51:30.634470    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037\": container with ID starting with 314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037 not found: ID does not exist" containerID="314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037"
Nov 02 12:51:30.634535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.634499    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037"} err="failed to get container status \"314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037\": rpc error: code = NotFound desc = could not find container \"314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037\": container with ID starting with 314554353f5e296c6981e0d07c2ed663b8fceda2637c62ac9ac49bd2ce09c037 not found: ID does not exist"
Nov 02 12:51:30.671445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.671416    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.671445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.671441    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.671445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.671449    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61768aba-42e4-4b0c-87e3-53c4e4e17c66-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.671640 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.671458    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbjsp\" (UniqueName: \"kubernetes.io/projected/61768aba-42e4-4b0c-87e3-53c4e4e17c66-kube-api-access-nbjsp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:30.675816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.675794    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"]
Nov 02 12:51:30.679896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.679867    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:30.692761 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.682556    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-729108/ipp-route-729108-kserve-7f65b4675c-w458s"]
Nov 02 12:51:30.767419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.767387    5879 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 02 12:51:30.767549 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.767429    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222" podUID="7036741f-e614-4dc6-b368-10e452e8310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 02 12:51:30.878276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:30.878233    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:31.628627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:31.628591    5879 generic.go:356] "Generic (PLEG): container finished" podID="912751e5-8e00-41f4-b45e-6a8244add550" containerID="64512a357070b08796d0e2f320480da0b3aa1df6fa5d94a607fdffa24952f2c0" exitCode=0
Nov 02 12:51:31.629205 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:31.628674    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n" event={"ID":"912751e5-8e00-41f4-b45e-6a8244add550","Type":"ContainerDied","Data":"64512a357070b08796d0e2f320480da0b3aa1df6fa5d94a607fdffa24952f2c0"}
Nov 02 12:51:31.767090 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:31.766978    5879 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 02 12:51:31.767090 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:31.767067    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222" podUID="7036741f-e614-4dc6-b368-10e452e8310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 02 12:51:32.018940 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:32.018869    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61768aba-42e4-4b0c-87e3-53c4e4e17c66" path="/var/lib/kubelet/pods/61768aba-42e4-4b0c-87e3-53c4e4e17c66/volumes"
Nov 02 12:51:32.055957 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:32.055923    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:32.766512 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:32.766477    5879 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 02 12:51:32.766915 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:32.766541    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222" podUID="7036741f-e614-4dc6-b368-10e452e8310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 02 12:51:32.827751 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:32.827726    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:32.932575 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:32.932485    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-openshift-ca\") pod \"912751e5-8e00-41f4-b45e-6a8244add550\" (UID: \"912751e5-8e00-41f4-b45e-6a8244add550\") "
Nov 02 12:51:32.932753 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:32.932701    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-xks-ca\") pod \"912751e5-8e00-41f4-b45e-6a8244add550\" (UID: \"912751e5-8e00-41f4-b45e-6a8244add550\") "
Nov 02 12:51:32.932848 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:32.932830    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912751e5-8e00-41f4-b45e-6a8244add550-kube-api-access-fllx6\" (UniqueName: \"kubernetes.io/projected/912751e5-8e00-41f4-b45e-6a8244add550-kube-api-access-fllx6\") pod \"912751e5-8e00-41f4-b45e-6a8244add550\" (UID: \"912751e5-8e00-41f4-b45e-6a8244add550\") "
Nov 02 12:51:32.932914 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:32.932858    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-openshift-ca" pod "912751e5-8e00-41f4-b45e-6a8244add550" (UID: "912751e5-8e00-41f4-b45e-6a8244add550"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:32.933424 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:32.933404    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:32.935670 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:32.935640    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-xks-ca" pod "912751e5-8e00-41f4-b45e-6a8244add550" (UID: "912751e5-8e00-41f4-b45e-6a8244add550"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:32.935797 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:32.935733    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912751e5-8e00-41f4-b45e-6a8244add550-kube-api-access-fllx6" pod "912751e5-8e00-41f4-b45e-6a8244add550" (UID: "912751e5-8e00-41f4-b45e-6a8244add550"). InnerVolumeSpecName "kube-api-access-fllx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:33.034875 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:33.034845    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912751e5-8e00-41f4-b45e-6a8244add550-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:33.034875 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:33.034873    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fllx6\" (UniqueName: \"kubernetes.io/projected/912751e5-8e00-41f4-b45e-6a8244add550-kube-api-access-fllx6\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:33.219750 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:33.219668    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:33.610323 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:33.610237    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"]
Nov 02 12:51:33.643668 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:33.643641    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"
Nov 02 12:51:33.643836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:33.643639    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n" event={"ID":"912751e5-8e00-41f4-b45e-6a8244add550","Type":"ContainerDied","Data":"650018e114590f30a30865fb87f45a280af6ec86ab122bf40068a739715a1938"}
Nov 02 12:51:33.643836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:33.643767    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="650018e114590f30a30865fb87f45a280af6ec86ab122bf40068a739715a1938"
Nov 02 12:51:33.766464 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:33.766432    5879 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 02 12:51:33.766625 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:33.766489    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222" podUID="7036741f-e614-4dc6-b368-10e452e8310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 02 12:51:33.806028 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:33.804329    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:34.044334 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:34.044280    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:34.766262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:34.766213    5879 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 02 12:51:34.766454 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:34.766265    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222" podUID="7036741f-e614-4dc6-b368-10e452e8310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 02 12:51:35.053585 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:35.053485    5879 scope.go:122] "RemoveContainer" containerID="793397b3a7edb0a22e228c14755d9473ea5dfe8ec4417ac6f3747fae16df4dec"
Nov 02 12:51:35.540918 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:35.540884    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:35.774017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:35.768920    5879 patch_prober.go:29] interesting pod/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Nov 02 12:51:35.774017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:35.769029    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222" podUID="7036741f-e614-4dc6-b368-10e452e8310a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Nov 02 12:51:36.002531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:36.002445    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:36.241142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:36.241112    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:36.376765 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:36.376720    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:36.771835 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:36.771762    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:36.772124 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:36.772109    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:36.772899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:36.772882    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:51:38.038668 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:38.038626    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:38.611536 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:38.611471    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs" podUID="f6ad669c-85d5-4c25-9e69-9348efb2d1ca" containerName="payload-pre-processing" containerID="cri-o://dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e" gracePeriod=25
Nov 02 12:51:38.883490 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:38.883460    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"
Nov 02 12:51:38.945964 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:38.945931    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-kube-api-access-89k8j\" (UniqueName: \"kubernetes.io/projected/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-kube-api-access-89k8j\") pod \"f6ad669c-85d5-4c25-9e69-9348efb2d1ca\" (UID: \"f6ad669c-85d5-4c25-9e69-9348efb2d1ca\") "
Nov 02 12:51:38.946237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:38.946221    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-plugins-config-volume\") pod \"f6ad669c-85d5-4c25-9e69-9348efb2d1ca\" (UID: \"f6ad669c-85d5-4c25-9e69-9348efb2d1ca\") "
Nov 02 12:51:38.946685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:38.946651    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-plugins-config-volume" pod "f6ad669c-85d5-4c25-9e69-9348efb2d1ca" (UID: "f6ad669c-85d5-4c25-9e69-9348efb2d1ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:38.948378 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:38.948345    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-kube-api-access-89k8j" pod "f6ad669c-85d5-4c25-9e69-9348efb2d1ca" (UID: "f6ad669c-85d5-4c25-9e69-9348efb2d1ca"). InnerVolumeSpecName "kube-api-access-89k8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:39.048443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.048409    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:39.048443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.048442    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89k8j\" (UniqueName: \"kubernetes.io/projected/f6ad669c-85d5-4c25-9e69-9348efb2d1ca-kube-api-access-89k8j\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:39.656185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.656151    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:39.710095 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.710062    5879 generic.go:356] "Generic (PLEG): container finished" podID="f6ad669c-85d5-4c25-9e69-9348efb2d1ca" containerID="dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e" exitCode=0
Nov 02 12:51:39.710264 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.710106    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs" event={"ID":"f6ad669c-85d5-4c25-9e69-9348efb2d1ca","Type":"ContainerDied","Data":"dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e"}
Nov 02 12:51:39.710264 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.710117    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"
Nov 02 12:51:39.710264 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.710148    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs" event={"ID":"f6ad669c-85d5-4c25-9e69-9348efb2d1ca","Type":"ContainerDied","Data":"45ebbdcb6ba6c22b4abf5653670ee5d87b0a2ea745a44147e89963e5fe9b00dc"}
Nov 02 12:51:39.710264 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.710172    5879 scope.go:122] "RemoveContainer" containerID="dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e"
Nov 02 12:51:39.721787 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.721767    5879 scope.go:122] "RemoveContainer" containerID="dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e"
Nov 02 12:51:39.722227 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:51:39.722208    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e\": container with ID starting with dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e not found: ID does not exist" containerID="dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e"
Nov 02 12:51:39.722307 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.722239    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e"} err="failed to get container status \"dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e\": rpc error: code = NotFound desc = could not find container \"dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e\": container with ID starting with dc47a7191077b5eb9465c4cab7f0e2dcbe507d44ae0f95276d6a1e32c29ba14e not found: ID does not exist"
Nov 02 12:51:39.771052 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.771016    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"]
Nov 02 12:51:39.782804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.782765    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9750c7-777968856c-59jjs"]
Nov 02 12:51:39.803141 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.803110    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:39.961215 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:39.961134    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:40.025257 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:40.025222    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6ad669c-85d5-4c25-9e69-9348efb2d1ca" path="/var/lib/kubelet/pods/f6ad669c-85d5-4c25-9e69-9348efb2d1ca/volumes"
Nov 02 12:51:40.430428 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:40.430393    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:42.314305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.314268    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:42.474783 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.474748    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:42.875533 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.875506    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"]
Nov 02 12:51:42.876065 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.876051    5879 cpu_manager.go:414] "containerMap: removing container" podUID="61768aba-42e4-4b0c-87e3-53c4e4e17c66" containerName="main"
Nov 02 12:51:42.876113 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.876065    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61768aba-42e4-4b0c-87e3-53c4e4e17c66" containerName="main"
Nov 02 12:51:42.876113 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.876090    5879 cpu_manager.go:414] "containerMap: removing container" podUID="912751e5-8e00-41f4-b45e-6a8244add550" containerName="revoke-keys"
Nov 02 12:51:42.876113 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.876095    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912751e5-8e00-41f4-b45e-6a8244add550" containerName="revoke-keys"
Nov 02 12:51:42.876113 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.876104    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f6ad669c-85d5-4c25-9e69-9348efb2d1ca" containerName="payload-pre-processing"
Nov 02 12:51:42.876113 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.876109    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6ad669c-85d5-4c25-9e69-9348efb2d1ca" containerName="payload-pre-processing"
Nov 02 12:51:42.876263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.876215    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="61768aba-42e4-4b0c-87e3-53c4e4e17c66" containerName="main"
Nov 02 12:51:42.876263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.876226    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="912751e5-8e00-41f4-b45e-6a8244add550" containerName="revoke-keys"
Nov 02 12:51:42.876263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.876234    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6ad669c-85d5-4c25-9e69-9348efb2d1ca" containerName="payload-pre-processing"
Nov 02 12:51:42.880827 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.880808    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:42.897256 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.897229    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:51:42.897256 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.897247    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:51:42.900695 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.899521    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"]
Nov 02 12:51:42.994092 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:42.994053    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:43.035198 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.035161    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aecfd915-ggxzz\" (UID: \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:43.035491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.035288    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfp4d\" (UniqueName: \"kubernetes.io/projected/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-kube-api-access-cfp4d\") pod \"maas-api-revoke-sub-aecfd915-ggxzz\" (UID: \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:43.035491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.035329    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aecfd915-ggxzz\" (UID: \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:43.136546 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.136454    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfp4d\" (UniqueName: \"kubernetes.io/projected/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-kube-api-access-cfp4d\") pod \"maas-api-revoke-sub-aecfd915-ggxzz\" (UID: \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:43.136546 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.136496    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aecfd915-ggxzz\" (UID: \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:43.136893 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.136855    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aecfd915-ggxzz\" (UID: \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:43.137455 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.137436    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aecfd915-ggxzz\" (UID: \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:43.139257 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.139237    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aecfd915-ggxzz\" (UID: \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:43.161849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.161818    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfp4d\" (UniqueName: \"kubernetes.io/projected/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-kube-api-access-cfp4d\") pod \"maas-api-revoke-sub-aecfd915-ggxzz\" (UID: \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:43.202525 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.202489    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:43.372322 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.372280    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:43.539428 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.539396    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"]
Nov 02 12:51:43.736904 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:43.736873    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz" event={"ID":"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac","Type":"ContainerStarted","Data":"3e3c36a5e5a1774b475bde0246bab90e407ff654943eb13c6a3e97312c260e14"}
Nov 02 12:51:44.777139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:44.777049    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:44.778293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:44.778255    5879 generic.go:356] "Generic (PLEG): container finished" podID="ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac" containerID="e4560a662aae08e0101ae15692ab2c7a10731639deaf866b6ce3dcaadec338c2" exitCode=0
Nov 02 12:51:44.778498 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:44.778438    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz" event={"ID":"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac","Type":"ContainerDied","Data":"e4560a662aae08e0101ae15692ab2c7a10731639deaf866b6ce3dcaadec338c2"}
Nov 02 12:51:44.832260 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:51:44.832224    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:51:44.998635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:44.998604    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"]
Nov 02 12:51:45.004086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.004053    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.027038 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.026982    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"]
Nov 02 12:51:45.173767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.173730    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.174045 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.174016    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.174185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.174169    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.174254 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.174195    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.174310 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.174288    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.174487 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.174470    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.174604 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.174585    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66efff16-b760-4e92-89de-c51ed4a48087-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.174884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.174860    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66efff16-b760-4e92-89de-c51ed4a48087-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.175139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.175119    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k77pm\" (UniqueName: \"kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-kube-api-access-k77pm\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.276801    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k77pm\" (UniqueName: \"kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-kube-api-access-k77pm\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.276903    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.276952    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.277058    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.277088    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.277147    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.277207    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.277232    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66efff16-b760-4e92-89de-c51ed4a48087-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.277396    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66efff16-b760-4e92-89de-c51ed4a48087-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.277900    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-certs\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.278048    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.278162    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-credential-socket\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.278354    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-data\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.279328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.278813    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66efff16-b760-4e92-89de-c51ed4a48087-istiod-ca-cert\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.281139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.281113    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-envoy\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.282531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.282509    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66efff16-b760-4e92-89de-c51ed4a48087-istio-podinfo\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.304411 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.304378    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-istio-token\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.304920 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.304892    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k77pm\" (UniqueName: \"kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-kube-api-access-k77pm\") pod \"e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") " pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.326873 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.326837    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:45.655192 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:51:45.655157    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66efff16_b760_4e92_89de_c51ed4a48087.slice/crio-13890ccd4681c1b0519ce55e4399de476daadba8ebd40ee3c34ee13e5acb3b25 WatchSource:0}: Error finding container 13890ccd4681c1b0519ce55e4399de476daadba8ebd40ee3c34ee13e5acb3b25: Status 404 returned error can't find the container with id 13890ccd4681c1b0519ce55e4399de476daadba8ebd40ee3c34ee13e5acb3b25
Nov 02 12:51:45.655695 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.655667    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"]
Nov 02 12:51:45.657916 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.657880    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:51:45.658055 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.657968    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:51:45.658055 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.658034    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:51:45.789084 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.789043    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" event={"ID":"66efff16-b760-4e92-89de-c51ed4a48087","Type":"ContainerStarted","Data":"663c3990db661a7dcd730c6591a857b7747bcf931ea3069288ee4f99fe718a9d"}
Nov 02 12:51:45.789594 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.789119    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" event={"ID":"66efff16-b760-4e92-89de-c51ed4a48087","Type":"ContainerStarted","Data":"13890ccd4681c1b0519ce55e4399de476daadba8ebd40ee3c34ee13e5acb3b25"}
Nov 02 12:51:45.843561 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.843504    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" podStartSLOduration=1.843488962 podStartE2EDuration="1.843488962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:45.842830403 +0000 UTC m=+1451.949533269" watchObservedRunningTime="2026-11-02 12:51:45.843488962 +0000 UTC m=+1451.950191813"
Nov 02 12:51:45.966523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.966497    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:45.987249 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.987211    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-xks-ca\") pod \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\" (UID: \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\") "
Nov 02 12:51:45.987463 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.987381    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-kube-api-access-cfp4d\" (UniqueName: \"kubernetes.io/projected/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-kube-api-access-cfp4d\") pod \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\" (UID: \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\") "
Nov 02 12:51:45.987463 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.987436    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-openshift-ca\") pod \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\" (UID: \"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac\") "
Nov 02 12:51:45.987876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.987845    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-openshift-ca" pod "ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac" (UID: "ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:45.988239 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.988217    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:45.989726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.989697    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-xks-ca" pod "ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac" (UID: "ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:45.990447 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:45.990425    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-kube-api-access-cfp4d" pod "ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac" (UID: "ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac"). InnerVolumeSpecName "kube-api-access-cfp4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:46.096432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:46.095982    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:46.096432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:46.096037    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfp4d\" (UniqueName: \"kubernetes.io/projected/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac-kube-api-access-cfp4d\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:46.160369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:46.160334    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:46.339036 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:46.335096    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:46.339036 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:46.335242    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 12:51:46.339036 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:46.335285    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" podUID="66efff16-b760-4e92-89de-c51ed4a48087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 12:51:46.813191 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:46.813158    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"
Nov 02 12:51:46.813608 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:46.813198    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz" event={"ID":"ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac","Type":"ContainerDied","Data":"3e3c36a5e5a1774b475bde0246bab90e407ff654943eb13c6a3e97312c260e14"}
Nov 02 12:51:46.813608 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:46.813251    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e3c36a5e5a1774b475bde0246bab90e407ff654943eb13c6a3e97312c260e14"
Nov 02 12:51:46.977917 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:46.977857    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:47.327707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:47.327670    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 12:51:47.327971 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:47.327724    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" podUID="66efff16-b760-4e92-89de-c51ed4a48087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 12:51:48.327391 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:48.327353    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 12:51:48.327870 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:48.327417    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" podUID="66efff16-b760-4e92-89de-c51ed4a48087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 12:51:49.065127 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:49.065084    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"]
Nov 02 12:51:49.065371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:49.065334    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4" podUID="344cbaff-2812-4910-a6dc-7c33fd030d39" containerName="istio-proxy" containerID="cri-o://31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c" gracePeriod=30
Nov 02 12:51:49.078431 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:49.078400    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"]
Nov 02 12:51:49.078687 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:49.078653    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w" podUID="1816f3b7-b324-45cd-add2-9931d7f1eb26" containerName="istio-proxy" containerID="cri-o://6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47" gracePeriod=30
Nov 02 12:51:49.272373 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:49.272334    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:49.330103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:49.329571    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 12:51:49.330103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:49.329633    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" podUID="66efff16-b760-4e92-89de-c51ed4a48087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 12:51:50.327640 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:50.327583    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 12:51:50.327841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:50.327643    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" podUID="66efff16-b760-4e92-89de-c51ed4a48087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 12:51:51.030398 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.030366    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:51.327980 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.327907    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 12:51:51.327980 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.327962    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" podUID="66efff16-b760-4e92-89de-c51ed4a48087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 12:51:51.530971 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.530934    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"]
Nov 02 12:51:51.531619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.531598    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac" containerName="revoke-keys"
Nov 02 12:51:51.531735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.531621    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac" containerName="revoke-keys"
Nov 02 12:51:51.531840 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.531826    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac" containerName="revoke-keys"
Nov 02 12:51:51.535334 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.535314    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:51.551826 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.551788    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:51:51.565353 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.565326    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:51:51.568377 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.568353    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"]
Nov 02 12:51:51.607344 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.607238    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snbcq\" (UniqueName: \"kubernetes.io/projected/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-kube-api-access-snbcq\") pod \"maas-api-revoke-sub-42499d2f-24bsz\" (UID: \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:51.607535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.607500    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42499d2f-24bsz\" (UID: \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:51.607611 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.607539    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42499d2f-24bsz\" (UID: \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:51.724111 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.719567    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snbcq\" (UniqueName: \"kubernetes.io/projected/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-kube-api-access-snbcq\") pod \"maas-api-revoke-sub-42499d2f-24bsz\" (UID: \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:51.724423 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.724394    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42499d2f-24bsz\" (UID: \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:51.724869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.724841    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42499d2f-24bsz\" (UID: \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:51.725392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.725345    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42499d2f-24bsz\" (UID: \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:51.733522 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.733459    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42499d2f-24bsz\" (UID: \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:51.743283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.743251    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:51.749326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.749289    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snbcq\" (UniqueName: \"kubernetes.io/projected/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-kube-api-access-snbcq\") pod \"maas-api-revoke-sub-42499d2f-24bsz\" (UID: \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:51.769496 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.769461    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"]
Nov 02 12:51:51.769700 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.769677    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" podUID="143c9cce-2a0f-454d-b363-87ebf116e254" containerName="istio-proxy" containerID="cri-o://da43298ba95d9f2323bfc806749b4e0d2e87fce74736868f9a51db4fea2b813c" gracePeriod=30
Nov 02 12:51:51.851018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:51.850259    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:52.198003 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.197892    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"]
Nov 02 12:51:52.202171 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.202151    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:52.239610 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.239565    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"]
Nov 02 12:51:52.258904 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.258868    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"]
Nov 02 12:51:52.259729 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:51:52.259690    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b9b75f_b1e5_4e8c_9989_c4d3059422d2.slice/crio-f08d1fedcb792fec8100f3eb4e9ab3b6f41b2ed3b50866d1a1788a50c2c2e64f WatchSource:0}: Error finding container f08d1fedcb792fec8100f3eb4e9ab3b6f41b2ed3b50866d1a1788a50c2c2e64f: Status 404 returned error can't find the container with id f08d1fedcb792fec8100f3eb4e9ab3b6f41b2ed3b50866d1a1788a50c2c2e64f
Nov 02 12:51:52.327903 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.327870    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 12:51:52.328101 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.327921    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" podUID="66efff16-b760-4e92-89de-c51ed4a48087" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 12:51:52.340920 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.340885    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fefe46e5-ftgfh\" (UID: \"f926c748-f9e9-4be9-9ae4-a06c61a76631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:52.341300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.341209    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fefe46e5-ftgfh\" (UID: \"f926c748-f9e9-4be9-9ae4-a06c61a76631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:52.341450 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.341434    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m49sc\" (UniqueName: \"kubernetes.io/projected/f926c748-f9e9-4be9-9ae4-a06c61a76631-kube-api-access-m49sc\") pod \"maas-api-revoke-sub-fefe46e5-ftgfh\" (UID: \"f926c748-f9e9-4be9-9ae4-a06c61a76631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:52.445139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.445094    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fefe46e5-ftgfh\" (UID: \"f926c748-f9e9-4be9-9ae4-a06c61a76631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:52.445353 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.445327    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m49sc\" (UniqueName: \"kubernetes.io/projected/f926c748-f9e9-4be9-9ae4-a06c61a76631-kube-api-access-m49sc\") pod \"maas-api-revoke-sub-fefe46e5-ftgfh\" (UID: \"f926c748-f9e9-4be9-9ae4-a06c61a76631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:52.445444 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.445428    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fefe46e5-ftgfh\" (UID: \"f926c748-f9e9-4be9-9ae4-a06c61a76631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:52.447944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.447914    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fefe46e5-ftgfh\" (UID: \"f926c748-f9e9-4be9-9ae4-a06c61a76631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:52.482618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.482534    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fefe46e5-ftgfh\" (UID: \"f926c748-f9e9-4be9-9ae4-a06c61a76631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:52.482793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.482737    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m49sc\" (UniqueName: \"kubernetes.io/projected/f926c748-f9e9-4be9-9ae4-a06c61a76631-kube-api-access-m49sc\") pod \"maas-api-revoke-sub-fefe46e5-ftgfh\" (UID: \"f926c748-f9e9-4be9-9ae4-a06c61a76631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:52.517737 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.517705    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:52.833060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.832920    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:52.840825 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.840796    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"]
Nov 02 12:51:52.878017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.866584    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz" event={"ID":"25b9b75f-b1e5-4e8c-9989-c4d3059422d2","Type":"ContainerStarted","Data":"5b249fc9d287b0d07dc964c9b462cc3e06cd3e30a1368470ef8a23d8001ff6ad"}
Nov 02 12:51:52.878017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.866670    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz" event={"ID":"25b9b75f-b1e5-4e8c-9989-c4d3059422d2","Type":"ContainerStarted","Data":"f08d1fedcb792fec8100f3eb4e9ab3b6f41b2ed3b50866d1a1788a50c2c2e64f"}
Nov 02 12:51:52.878017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:52.869676    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh" event={"ID":"f926c748-f9e9-4be9-9ae4-a06c61a76631","Type":"ContainerStarted","Data":"706b7cde206274f69c9d7386431f75add7af2edcb2a424d39c03a10577846d9a"}
Nov 02 12:51:53.016439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.016167    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz" podStartSLOduration=2.016053146 podStartE2EDuration="2.016053146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:52.939422081 +0000 UTC m=+1459.046124932" watchObservedRunningTime="2026-11-02 12:51:53.016053146 +0000 UTC m=+1459.122755999"
Nov 02 12:51:53.045102 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.045036    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:53.375728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.375697    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"]
Nov 02 12:51:53.381657 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.381634    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:53.400915 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.400886    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"]
Nov 02 12:51:53.480750 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.480715    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24a532c0-lfvvd\" (UID: \"07b7068f-e026-4857-bac2-bd487fca458e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:53.480906 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.480885    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p9t4\" (UniqueName: \"kubernetes.io/projected/07b7068f-e026-4857-bac2-bd487fca458e-kube-api-access-8p9t4\") pod \"maas-api-revoke-sub-24a532c0-lfvvd\" (UID: \"07b7068f-e026-4857-bac2-bd487fca458e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:53.481010 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.480947    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24a532c0-lfvvd\" (UID: \"07b7068f-e026-4857-bac2-bd487fca458e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:53.582810 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.582717    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24a532c0-lfvvd\" (UID: \"07b7068f-e026-4857-bac2-bd487fca458e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:53.582975 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.582953    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24a532c0-lfvvd\" (UID: \"07b7068f-e026-4857-bac2-bd487fca458e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:53.583134 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.583116    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p9t4\" (UniqueName: \"kubernetes.io/projected/07b7068f-e026-4857-bac2-bd487fca458e-kube-api-access-8p9t4\") pod \"maas-api-revoke-sub-24a532c0-lfvvd\" (UID: \"07b7068f-e026-4857-bac2-bd487fca458e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:53.583770 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.583743    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24a532c0-lfvvd\" (UID: \"07b7068f-e026-4857-bac2-bd487fca458e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:53.585322 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.585290    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24a532c0-lfvvd\" (UID: \"07b7068f-e026-4857-bac2-bd487fca458e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:53.609728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.609697    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p9t4\" (UniqueName: \"kubernetes.io/projected/07b7068f-e026-4857-bac2-bd487fca458e-kube-api-access-8p9t4\") pod \"maas-api-revoke-sub-24a532c0-lfvvd\" (UID: \"07b7068f-e026-4857-bac2-bd487fca458e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:53.691626 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.691591    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:53.692137 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.692112    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:53.693035 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.693015    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:51:53.703396 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.703372    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:53.826983 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.826932    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 12:51:53.827228 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.827070    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w" podUID="1816f3b7-b324-45cd-add2-9931d7f1eb26" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 12:51:53.879053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.879015    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh" event={"ID":"f926c748-f9e9-4be9-9ae4-a06c61a76631","Type":"ContainerStarted","Data":"e025a956cb4cc9fa463557d8f24ee173ea81d195d5a69036ac0c446e46388edb"}
Nov 02 12:51:53.881284 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.881253    5879 generic.go:356] "Generic (PLEG): container finished" podID="25b9b75f-b1e5-4e8c-9989-c4d3059422d2" containerID="5b249fc9d287b0d07dc964c9b462cc3e06cd3e30a1368470ef8a23d8001ff6ad" exitCode=0
Nov 02 12:51:53.881476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.881340    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz" event={"ID":"25b9b75f-b1e5-4e8c-9989-c4d3059422d2","Type":"ContainerDied","Data":"5b249fc9d287b0d07dc964c9b462cc3e06cd3e30a1368470ef8a23d8001ff6ad"}
Nov 02 12:51:53.926477 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:53.926413    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh" podStartSLOduration=1.9263935110000001 podStartE2EDuration="1.926393511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:53.922191369 +0000 UTC m=+1460.028894220" watchObservedRunningTime="2026-11-02 12:51:53.926393511 +0000 UTC m=+1460.033096363"
Nov 02 12:51:54.020380 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.020351    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"]
Nov 02 12:51:54.020603 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:51:54.020581    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b7068f_e026_4857_bac2_bd487fca458e.slice/crio-412ded0dd0aaf522c6c6d08b6b34203283e8392333c910b858d8f199cb2bd9d9 WatchSource:0}: Error finding container 412ded0dd0aaf522c6c6d08b6b34203283e8392333c910b858d8f199cb2bd9d9: Status 404 returned error can't find the container with id 412ded0dd0aaf522c6c6d08b6b34203283e8392333c910b858d8f199cb2bd9d9
Nov 02 12:51:54.399933 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.399902    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:51:54.486432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.486404    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:51:54.504924 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.504877    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/344cbaff-2812-4910-a6dc-7c33fd030d39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344cbaff-2812-4910-a6dc-7c33fd030d39-istiod-ca-cert\") pod \"344cbaff-2812-4910-a6dc-7c33fd030d39\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") "
Nov 02 12:51:54.505110 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505054    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-certs\") pod \"344cbaff-2812-4910-a6dc-7c33fd030d39\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") "
Nov 02 12:51:54.505110 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505099    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-data\") pod \"344cbaff-2812-4910-a6dc-7c33fd030d39\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") "
Nov 02 12:51:54.505294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505276    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-credential-socket\") pod \"344cbaff-2812-4910-a6dc-7c33fd030d39\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") "
Nov 02 12:51:54.505370 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505316    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-podinfo\") pod \"344cbaff-2812-4910-a6dc-7c33fd030d39\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") "
Nov 02 12:51:54.505370 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505341    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-kube-api-access-gsc4m\" (UniqueName: \"kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-kube-api-access-gsc4m\") pod \"344cbaff-2812-4910-a6dc-7c33fd030d39\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") "
Nov 02 12:51:54.505517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505408    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-envoy\") pod \"344cbaff-2812-4910-a6dc-7c33fd030d39\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") "
Nov 02 12:51:54.505517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505344    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/344cbaff-2812-4910-a6dc-7c33fd030d39-istiod-ca-cert" pod "344cbaff-2812-4910-a6dc-7c33fd030d39" (UID: "344cbaff-2812-4910-a6dc-7c33fd030d39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:54.505517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505358    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-certs" pod "344cbaff-2812-4910-a6dc-7c33fd030d39" (UID: "344cbaff-2812-4910-a6dc-7c33fd030d39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:54.505517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505422    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-data" pod "344cbaff-2812-4910-a6dc-7c33fd030d39" (UID: "344cbaff-2812-4910-a6dc-7c33fd030d39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:54.505517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505498    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-token\" (UniqueName: \"kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-token\") pod \"344cbaff-2812-4910-a6dc-7c33fd030d39\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") "
Nov 02 12:51:54.505773 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505553    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-socket\") pod \"344cbaff-2812-4910-a6dc-7c33fd030d39\" (UID: \"344cbaff-2812-4910-a6dc-7c33fd030d39\") "
Nov 02 12:51:54.505773 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505584    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-credential-socket" pod "344cbaff-2812-4910-a6dc-7c33fd030d39" (UID: "344cbaff-2812-4910-a6dc-7c33fd030d39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:54.505966 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.505909    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-socket" pod "344cbaff-2812-4910-a6dc-7c33fd030d39" (UID: "344cbaff-2812-4910-a6dc-7c33fd030d39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:54.506374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.506295    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.506374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.506313    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.506374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.506323    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.506374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.506337    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.506374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.506347    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/344cbaff-2812-4910-a6dc-7c33fd030d39-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.507900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.507877    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-envoy" pod "344cbaff-2812-4910-a6dc-7c33fd030d39" (UID: "344cbaff-2812-4910-a6dc-7c33fd030d39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:54.508275 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.508253    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-podinfo" pod "344cbaff-2812-4910-a6dc-7c33fd030d39" (UID: "344cbaff-2812-4910-a6dc-7c33fd030d39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:51:54.508337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.508293    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-token" pod "344cbaff-2812-4910-a6dc-7c33fd030d39" (UID: "344cbaff-2812-4910-a6dc-7c33fd030d39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:54.508337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.508314    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-kube-api-access-gsc4m" pod "344cbaff-2812-4910-a6dc-7c33fd030d39" (UID: "344cbaff-2812-4910-a6dc-7c33fd030d39"). InnerVolumeSpecName "kube-api-access-gsc4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:54.607205 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607171    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1816f3b7-b324-45cd-add2-9931d7f1eb26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1816f3b7-b324-45cd-add2-9931d7f1eb26-istiod-ca-cert\") pod \"1816f3b7-b324-45cd-add2-9931d7f1eb26\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") "
Nov 02 12:51:54.607394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607290    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-credential-socket\") pod \"1816f3b7-b324-45cd-add2-9931d7f1eb26\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") "
Nov 02 12:51:54.607394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607393    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-envoy\") pod \"1816f3b7-b324-45cd-add2-9931d7f1eb26\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") "
Nov 02 12:51:54.607495 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607431    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-podinfo\") pod \"1816f3b7-b324-45cd-add2-9931d7f1eb26\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") "
Nov 02 12:51:54.607538 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607490    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-socket\") pod \"1816f3b7-b324-45cd-add2-9931d7f1eb26\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") "
Nov 02 12:51:54.607579 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607536    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-data\") pod \"1816f3b7-b324-45cd-add2-9931d7f1eb26\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") "
Nov 02 12:51:54.607619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607572    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-credential-socket" pod "1816f3b7-b324-45cd-add2-9931d7f1eb26" (UID: "1816f3b7-b324-45cd-add2-9931d7f1eb26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:54.607619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607610    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-certs\") pod \"1816f3b7-b324-45cd-add2-9931d7f1eb26\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") "
Nov 02 12:51:54.607695 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607615    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1816f3b7-b324-45cd-add2-9931d7f1eb26-istiod-ca-cert" pod "1816f3b7-b324-45cd-add2-9931d7f1eb26" (UID: "1816f3b7-b324-45cd-add2-9931d7f1eb26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:54.607695 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607644    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-kube-api-access-9thv5\" (UniqueName: \"kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-kube-api-access-9thv5\") pod \"1816f3b7-b324-45cd-add2-9931d7f1eb26\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") "
Nov 02 12:51:54.607776 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607706    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-token\" (UniqueName: \"kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-token\") pod \"1816f3b7-b324-45cd-add2-9931d7f1eb26\" (UID: \"1816f3b7-b324-45cd-add2-9931d7f1eb26\") "
Nov 02 12:51:54.607812 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607787    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-socket" pod "1816f3b7-b324-45cd-add2-9931d7f1eb26" (UID: "1816f3b7-b324-45cd-add2-9931d7f1eb26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:54.608146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.607933    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-certs" pod "1816f3b7-b324-45cd-add2-9931d7f1eb26" (UID: "1816f3b7-b324-45cd-add2-9931d7f1eb26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:54.608146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.608113    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-data" pod "1816f3b7-b324-45cd-add2-9931d7f1eb26" (UID: "1816f3b7-b324-45cd-add2-9931d7f1eb26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:54.608687 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.608666    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.608789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.608694    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.608789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.608709    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.608789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.608722    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsc4m\" (UniqueName: \"kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-kube-api-access-gsc4m\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.608789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.608748    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.608789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.608762    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/344cbaff-2812-4910-a6dc-7c33fd030d39-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.608789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.608774    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1816f3b7-b324-45cd-add2-9931d7f1eb26-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.608789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.608786    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.609215 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.608798    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.609964 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.609939    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-podinfo" pod "1816f3b7-b324-45cd-add2-9931d7f1eb26" (UID: "1816f3b7-b324-45cd-add2-9931d7f1eb26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:51:54.609964 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.609956    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-envoy" pod "1816f3b7-b324-45cd-add2-9931d7f1eb26" (UID: "1816f3b7-b324-45cd-add2-9931d7f1eb26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:54.610242 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.610224    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-kube-api-access-9thv5" pod "1816f3b7-b324-45cd-add2-9931d7f1eb26" (UID: "1816f3b7-b324-45cd-add2-9931d7f1eb26"). InnerVolumeSpecName "kube-api-access-9thv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:54.610347 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.610297    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-token" pod "1816f3b7-b324-45cd-add2-9931d7f1eb26" (UID: "1816f3b7-b324-45cd-add2-9931d7f1eb26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:54.710345 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.710313    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.710345 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.710342    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.710345 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.710351    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9thv5\" (UniqueName: \"kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-kube-api-access-9thv5\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.710589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.710361    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1816f3b7-b324-45cd-add2-9931d7f1eb26-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:54.770259 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.770220    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:54.891051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.891003    5879 generic.go:356] "Generic (PLEG): container finished" podID="344cbaff-2812-4910-a6dc-7c33fd030d39" containerID="31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c" exitCode=0
Nov 02 12:51:54.891051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.891047    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4" event={"ID":"344cbaff-2812-4910-a6dc-7c33fd030d39","Type":"ContainerDied","Data":"31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c"}
Nov 02 12:51:54.891329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.891085    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"
Nov 02 12:51:54.891329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.891105    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4" event={"ID":"344cbaff-2812-4910-a6dc-7c33fd030d39","Type":"ContainerDied","Data":"0cd74b774118ccf4810cb042d5b3ebfa0130fbfc61ce0c2ca6c1507e2c2c2cf1"}
Nov 02 12:51:54.891329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.891127    5879 scope.go:122] "RemoveContainer" containerID="31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c"
Nov 02 12:51:54.894720 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.894690    5879 generic.go:356] "Generic (PLEG): container finished" podID="f926c748-f9e9-4be9-9ae4-a06c61a76631" containerID="e025a956cb4cc9fa463557d8f24ee173ea81d195d5a69036ac0c446e46388edb" exitCode=0
Nov 02 12:51:54.894884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.894865    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh" event={"ID":"f926c748-f9e9-4be9-9ae4-a06c61a76631","Type":"ContainerDied","Data":"e025a956cb4cc9fa463557d8f24ee173ea81d195d5a69036ac0c446e46388edb"}
Nov 02 12:51:54.896857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.896832    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd" event={"ID":"07b7068f-e026-4857-bac2-bd487fca458e","Type":"ContainerStarted","Data":"b422bf728d761e2d9912ca166b7e327debaf716d84c1bc441bfe296f0fb59280"}
Nov 02 12:51:54.897015 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.896882    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd" event={"ID":"07b7068f-e026-4857-bac2-bd487fca458e","Type":"ContainerStarted","Data":"412ded0dd0aaf522c6c6d08b6b34203283e8392333c910b858d8f199cb2bd9d9"}
Nov 02 12:51:54.898565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.898543    5879 generic.go:356] "Generic (PLEG): container finished" podID="1816f3b7-b324-45cd-add2-9931d7f1eb26" containerID="6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47" exitCode=0
Nov 02 12:51:54.898705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.898594    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"
Nov 02 12:51:54.898705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.898632    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w" event={"ID":"1816f3b7-b324-45cd-add2-9931d7f1eb26","Type":"ContainerDied","Data":"6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47"}
Nov 02 12:51:54.898705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.898669    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w" event={"ID":"1816f3b7-b324-45cd-add2-9931d7f1eb26","Type":"ContainerDied","Data":"2b40c32a5ad85f8233bcbf09212a59d77d04a27d9514d5298f501a97934df3f7"}
Nov 02 12:51:54.904685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.904668    5879 scope.go:122] "RemoveContainer" containerID="31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c"
Nov 02 12:51:54.904978 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:51:54.904960    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c\": container with ID starting with 31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c not found: ID does not exist" containerID="31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c"
Nov 02 12:51:54.905062 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.904986    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c"} err="failed to get container status \"31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c\": rpc error: code = NotFound desc = could not find container \"31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c\": container with ID starting with 31b02752bf5b52cbee712df7fc8cf60d48706ce7949b3f3db2b8b6ec3f29a43c not found: ID does not exist"
Nov 02 12:51:54.905136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.905069    5879 scope.go:122] "RemoveContainer" containerID="6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47"
Nov 02 12:51:54.916627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.916607    5879 scope.go:122] "RemoveContainer" containerID="6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47"
Nov 02 12:51:54.916930 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:51:54.916899    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47\": container with ID starting with 6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47 not found: ID does not exist" containerID="6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47"
Nov 02 12:51:54.917020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.916937    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47"} err="failed to get container status \"6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47\": rpc error: code = NotFound desc = could not find container \"6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47\": container with ID starting with 6794ce1b5914f374dd3d93bfb94c999ed463778ef739e0b9571489f9c0b2cc47 not found: ID does not exist"
Nov 02 12:51:54.952691 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.952634    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd" podStartSLOduration=1.9526210609999999 podStartE2EDuration="1.952621061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:51:54.951179287 +0000 UTC m=+1461.057882128" watchObservedRunningTime="2026-11-02 12:51:54.952621061 +0000 UTC m=+1461.059323911"
Nov 02 12:51:55.002110 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:54.996654    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"]
Nov 02 12:51:55.002110 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.001726    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-s6fz4"]
Nov 02 12:51:55.071635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.071610    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:55.096427 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.096362    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"]
Nov 02 12:51:55.100044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.100007    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-75v6w"]
Nov 02 12:51:55.218315 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.218282    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-openshift-ca\") pod \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\" (UID: \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\") "
Nov 02 12:51:55.218505 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.218396    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-xks-ca\") pod \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\" (UID: \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\") "
Nov 02 12:51:55.218505 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.218467    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-kube-api-access-snbcq\" (UniqueName: \"kubernetes.io/projected/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-kube-api-access-snbcq\") pod \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\" (UID: \"25b9b75f-b1e5-4e8c-9989-c4d3059422d2\") "
Nov 02 12:51:55.218728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.218694    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-openshift-ca" pod "25b9b75f-b1e5-4e8c-9989-c4d3059422d2" (UID: "25b9b75f-b1e5-4e8c-9989-c4d3059422d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:55.218962 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.218949    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:55.220704 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.220680    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-xks-ca" pod "25b9b75f-b1e5-4e8c-9989-c4d3059422d2" (UID: "25b9b75f-b1e5-4e8c-9989-c4d3059422d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:55.220793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.220738    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-kube-api-access-snbcq" pod "25b9b75f-b1e5-4e8c-9989-c4d3059422d2" (UID: "25b9b75f-b1e5-4e8c-9989-c4d3059422d2"). InnerVolumeSpecName "kube-api-access-snbcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:55.320078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.319967    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:55.320078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.320018    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snbcq\" (UniqueName: \"kubernetes.io/projected/25b9b75f-b1e5-4e8c-9989-c4d3059422d2-kube-api-access-snbcq\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:55.907237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.907206    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"
Nov 02 12:51:55.907699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.907202    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz" event={"ID":"25b9b75f-b1e5-4e8c-9989-c4d3059422d2","Type":"ContainerDied","Data":"f08d1fedcb792fec8100f3eb4e9ab3b6f41b2ed3b50866d1a1788a50c2c2e64f"}
Nov 02 12:51:55.907699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.907328    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f08d1fedcb792fec8100f3eb4e9ab3b6f41b2ed3b50866d1a1788a50c2c2e64f"
Nov 02 12:51:55.910631 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.910598    5879 generic.go:356] "Generic (PLEG): container finished" podID="07b7068f-e026-4857-bac2-bd487fca458e" containerID="b422bf728d761e2d9912ca166b7e327debaf716d84c1bc441bfe296f0fb59280" exitCode=0
Nov 02 12:51:55.910820 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:55.910675    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd" event={"ID":"07b7068f-e026-4857-bac2-bd487fca458e","Type":"ContainerDied","Data":"b422bf728d761e2d9912ca166b7e327debaf716d84c1bc441bfe296f0fb59280"}
Nov 02 12:51:56.022958 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.022919    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1816f3b7-b324-45cd-add2-9931d7f1eb26" path="/var/lib/kubelet/pods/1816f3b7-b324-45cd-add2-9931d7f1eb26/volumes"
Nov 02 12:51:56.023737 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.023710    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="344cbaff-2812-4910-a6dc-7c33fd030d39" path="/var/lib/kubelet/pods/344cbaff-2812-4910-a6dc-7c33fd030d39/volumes"
Nov 02 12:51:56.092280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.092238    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:56.131026 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.130956    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-xks-ca\") pod \"f926c748-f9e9-4be9-9ae4-a06c61a76631\" (UID: \"f926c748-f9e9-4be9-9ae4-a06c61a76631\") "
Nov 02 12:51:56.131233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.131089    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f926c748-f9e9-4be9-9ae4-a06c61a76631-kube-api-access-m49sc\" (UniqueName: \"kubernetes.io/projected/f926c748-f9e9-4be9-9ae4-a06c61a76631-kube-api-access-m49sc\") pod \"f926c748-f9e9-4be9-9ae4-a06c61a76631\" (UID: \"f926c748-f9e9-4be9-9ae4-a06c61a76631\") "
Nov 02 12:51:56.131233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.131170    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-openshift-ca\") pod \"f926c748-f9e9-4be9-9ae4-a06c61a76631\" (UID: \"f926c748-f9e9-4be9-9ae4-a06c61a76631\") "
Nov 02 12:51:56.131522 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.131486    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-openshift-ca" pod "f926c748-f9e9-4be9-9ae4-a06c61a76631" (UID: "f926c748-f9e9-4be9-9ae4-a06c61a76631"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:56.131868 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.131687    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:56.133567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.133543    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-xks-ca" pod "f926c748-f9e9-4be9-9ae4-a06c61a76631" (UID: "f926c748-f9e9-4be9-9ae4-a06c61a76631"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:56.133669 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.133545    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f926c748-f9e9-4be9-9ae4-a06c61a76631-kube-api-access-m49sc" pod "f926c748-f9e9-4be9-9ae4-a06c61a76631" (UID: "f926c748-f9e9-4be9-9ae4-a06c61a76631"). InnerVolumeSpecName "kube-api-access-m49sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:56.232738 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.232658    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m49sc\" (UniqueName: \"kubernetes.io/projected/f926c748-f9e9-4be9-9ae4-a06c61a76631-kube-api-access-m49sc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:56.232738 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.232688    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f926c748-f9e9-4be9-9ae4-a06c61a76631-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:56.500429 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.500349    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:56.764844 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.764762    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:56.896368 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.896338    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:56.922229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.922197    5879 generic.go:356] "Generic (PLEG): container finished" podID="143c9cce-2a0f-454d-b363-87ebf116e254" containerID="da43298ba95d9f2323bfc806749b4e0d2e87fce74736868f9a51db4fea2b813c" exitCode=0
Nov 02 12:51:56.922753 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.922279    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" event={"ID":"143c9cce-2a0f-454d-b363-87ebf116e254","Type":"ContainerDied","Data":"da43298ba95d9f2323bfc806749b4e0d2e87fce74736868f9a51db4fea2b813c"}
Nov 02 12:51:56.924106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.924043    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"
Nov 02 12:51:56.924106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.924045    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh" event={"ID":"f926c748-f9e9-4be9-9ae4-a06c61a76631","Type":"ContainerDied","Data":"706b7cde206274f69c9d7386431f75add7af2edcb2a424d39c03a10577846d9a"}
Nov 02 12:51:56.924106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:56.924092    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="706b7cde206274f69c9d7386431f75add7af2edcb2a424d39c03a10577846d9a"
Nov 02 12:51:57.044054 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.043078    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:51:57.120599 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.120576    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:57.145733 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.145704    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/143c9cce-2a0f-454d-b363-87ebf116e254-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/143c9cce-2a0f-454d-b363-87ebf116e254-istio-podinfo\") pod \"143c9cce-2a0f-454d-b363-87ebf116e254\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") "
Nov 02 12:51:57.145919 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.145778    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143c9cce-2a0f-454d-b363-87ebf116e254-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/143c9cce-2a0f-454d-b363-87ebf116e254-istiod-ca-cert\") pod \"143c9cce-2a0f-454d-b363-87ebf116e254\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") "
Nov 02 12:51:57.145919 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.145886    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-credential-socket\") pod \"143c9cce-2a0f-454d-b363-87ebf116e254\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") "
Nov 02 12:51:57.146134 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.145966    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-socket\") pod \"143c9cce-2a0f-454d-b363-87ebf116e254\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") "
Nov 02 12:51:57.146134 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146029    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-certs\") pod \"143c9cce-2a0f-454d-b363-87ebf116e254\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") "
Nov 02 12:51:57.146134 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146099    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b7068f-e026-4857-bac2-bd487fca458e-kube-api-access-8p9t4\" (UniqueName: \"kubernetes.io/projected/07b7068f-e026-4857-bac2-bd487fca458e-kube-api-access-8p9t4\") pod \"07b7068f-e026-4857-bac2-bd487fca458e\" (UID: \"07b7068f-e026-4857-bac2-bd487fca458e\") "
Nov 02 12:51:57.146295 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146158    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-data\") pod \"143c9cce-2a0f-454d-b363-87ebf116e254\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") "
Nov 02 12:51:57.146295 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146196    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-envoy\") pod \"143c9cce-2a0f-454d-b363-87ebf116e254\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") "
Nov 02 12:51:57.146399 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146227    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143c9cce-2a0f-454d-b363-87ebf116e254-istiod-ca-cert" pod "143c9cce-2a0f-454d-b363-87ebf116e254" (UID: "143c9cce-2a0f-454d-b363-87ebf116e254"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:57.146399 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146301    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-credential-socket" pod "143c9cce-2a0f-454d-b363-87ebf116e254" (UID: "143c9cce-2a0f-454d-b363-87ebf116e254"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:57.146399 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146296    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-openshift-ca\") pod \"07b7068f-e026-4857-bac2-bd487fca458e\" (UID: \"07b7068f-e026-4857-bac2-bd487fca458e\") "
Nov 02 12:51:57.146557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146397    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-istio-token\" (UniqueName: \"kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-istio-token\") pod \"143c9cce-2a0f-454d-b363-87ebf116e254\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") "
Nov 02 12:51:57.146557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146416    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-certs" pod "143c9cce-2a0f-454d-b363-87ebf116e254" (UID: "143c9cce-2a0f-454d-b363-87ebf116e254"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:57.146557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146456    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-xks-ca\") pod \"07b7068f-e026-4857-bac2-bd487fca458e\" (UID: \"07b7068f-e026-4857-bac2-bd487fca458e\") "
Nov 02 12:51:57.146557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146499    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-kube-api-access-mbjwk\" (UniqueName: \"kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-kube-api-access-mbjwk\") pod \"143c9cce-2a0f-454d-b363-87ebf116e254\" (UID: \"143c9cce-2a0f-454d-b363-87ebf116e254\") "
Nov 02 12:51:57.146557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146547    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-data" pod "143c9cce-2a0f-454d-b363-87ebf116e254" (UID: "143c9cce-2a0f-454d-b363-87ebf116e254"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:57.146767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146629    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-openshift-ca" pod "07b7068f-e026-4857-bac2-bd487fca458e" (UID: "07b7068f-e026-4857-bac2-bd487fca458e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:51:57.146767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.146640    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-socket" pod "143c9cce-2a0f-454d-b363-87ebf116e254" (UID: "143c9cce-2a0f-454d-b363-87ebf116e254"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:57.147214 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.147187    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:57.147214 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.147209    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/143c9cce-2a0f-454d-b363-87ebf116e254-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:57.147350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.147224    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:57.147350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.147236    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:57.147350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.147247    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:57.147350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.147259    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:57.151304 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.151207    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/143c9cce-2a0f-454d-b363-87ebf116e254-istio-podinfo" pod "143c9cce-2a0f-454d-b363-87ebf116e254" (UID: "143c9cce-2a0f-454d-b363-87ebf116e254"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:51:57.151479 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.151454    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b7068f-e026-4857-bac2-bd487fca458e-kube-api-access-8p9t4" pod "07b7068f-e026-4857-bac2-bd487fca458e" (UID: "07b7068f-e026-4857-bac2-bd487fca458e"). InnerVolumeSpecName "kube-api-access-8p9t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:57.152019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.151957    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-kube-api-access-mbjwk" pod "143c9cce-2a0f-454d-b363-87ebf116e254" (UID: "143c9cce-2a0f-454d-b363-87ebf116e254"). InnerVolumeSpecName "kube-api-access-mbjwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:57.152335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.152313    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-envoy" pod "143c9cce-2a0f-454d-b363-87ebf116e254" (UID: "143c9cce-2a0f-454d-b363-87ebf116e254"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:51:57.153429 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.153402    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-xks-ca" pod "07b7068f-e026-4857-bac2-bd487fca458e" (UID: "07b7068f-e026-4857-bac2-bd487fca458e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:51:57.153670 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.153649    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-istio-token" pod "143c9cce-2a0f-454d-b363-87ebf116e254" (UID: "143c9cce-2a0f-454d-b363-87ebf116e254"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:51:57.249033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.248980    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p9t4\" (UniqueName: \"kubernetes.io/projected/07b7068f-e026-4857-bac2-bd487fca458e-kube-api-access-8p9t4\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:57.249033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.249030    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/143c9cce-2a0f-454d-b363-87ebf116e254-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:57.249033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.249039    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:57.249280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.249047    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b7068f-e026-4857-bac2-bd487fca458e-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:57.249280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.249055    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbjwk\" (UniqueName: \"kubernetes.io/projected/143c9cce-2a0f-454d-b363-87ebf116e254-kube-api-access-mbjwk\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:57.249280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.249063    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/143c9cce-2a0f-454d-b363-87ebf116e254-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:51:57.541043 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.541011    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:57.755808 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.755768    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:57.965023 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.956906    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:57.973085 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.967594    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l" event={"ID":"143c9cce-2a0f-454d-b363-87ebf116e254","Type":"ContainerDied","Data":"c1cbd65f243e3ff54565f4b4fd2a79174c936a841edaab034e1b43063cfc667a"}
Nov 02 12:51:57.973085 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.967669    5879 scope.go:122] "RemoveContainer" containerID="da43298ba95d9f2323bfc806749b4e0d2e87fce74736868f9a51db4fea2b813c"
Nov 02 12:51:57.973085 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:57.967825    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"
Nov 02 12:51:58.029021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:58.024096    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"]
Nov 02 12:51:58.029021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:58.024350    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv" podUID="ec0c26de-cbe1-462a-9d0c-ada687f84a78" containerName="istio-proxy" containerID="cri-o://8f145e506bd40b11daa786109cb081e838c7563fe81e0f12ddb213eab90dc6ac" gracePeriod=30
Nov 02 12:51:58.083021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:58.060869    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"
Nov 02 12:51:58.091332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:58.091082    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"]
Nov 02 12:51:58.091332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:58.091159    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd" event={"ID":"07b7068f-e026-4857-bac2-bd487fca458e","Type":"ContainerDied","Data":"412ded0dd0aaf522c6c6d08b6b34203283e8392333c910b858d8f199cb2bd9d9"}
Nov 02 12:51:58.091332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:58.091197    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="412ded0dd0aaf522c6c6d08b6b34203283e8392333c910b858d8f199cb2bd9d9"
Nov 02 12:51:58.091671 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:58.091373    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222" podUID="7036741f-e614-4dc6-b368-10e452e8310a" containerName="istio-proxy" containerID="cri-o://ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659" gracePeriod=30
Nov 02 12:51:58.099534 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:58.099502    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:58.179669 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:58.179602    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"]
Nov 02 12:51:58.179944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:58.179926    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-ckc4l"]
Nov 02 12:51:59.320018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:59.319592    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:51:59.580272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:51:59.572806    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:00.017200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:00.017170    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143c9cce-2a0f-454d-b363-87ebf116e254" path="/var/lib/kubelet/pods/143c9cce-2a0f-454d-b363-87ebf116e254/volumes"
Nov 02 12:52:01.759890 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:01.759860    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:03.203879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.203842    5879 generic.go:356] "Generic (PLEG): container finished" podID="ec0c26de-cbe1-462a-9d0c-ada687f84a78" containerID="8f145e506bd40b11daa786109cb081e838c7563fe81e0f12ddb213eab90dc6ac" exitCode=0
Nov 02 12:52:03.204483 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.203921    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv" event={"ID":"ec0c26de-cbe1-462a-9d0c-ada687f84a78","Type":"ContainerDied","Data":"8f145e506bd40b11daa786109cb081e838c7563fe81e0f12ddb213eab90dc6ac"}
Nov 02 12:52:03.292677 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.292652    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:52:03.381630 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.381598    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-envoy\") pod \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") "
Nov 02 12:52:03.381793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.381713    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-credential-socket\") pod \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") "
Nov 02 12:52:03.381793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.381765    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-certs\") pod \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") "
Nov 02 12:52:03.381915 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.381814    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istiod-ca-cert\") pod \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") "
Nov 02 12:52:03.381974 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.381911    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-kube-api-access-lxpcc\" (UniqueName: \"kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-kube-api-access-lxpcc\") pod \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") "
Nov 02 12:52:03.381974 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.381959    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-data\") pod \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") "
Nov 02 12:52:03.382866 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.382068    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-token\" (UniqueName: \"kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-token\") pod \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") "
Nov 02 12:52:03.382866 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.382115    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-socket\") pod \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") "
Nov 02 12:52:03.382866 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.382623    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-certs" pod "ec0c26de-cbe1-462a-9d0c-ada687f84a78" (UID: "ec0c26de-cbe1-462a-9d0c-ada687f84a78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.382866 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.382648    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-data" pod "ec0c26de-cbe1-462a-9d0c-ada687f84a78" (UID: "ec0c26de-cbe1-462a-9d0c-ada687f84a78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.382866 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.382727    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-credential-socket" pod "ec0c26de-cbe1-462a-9d0c-ada687f84a78" (UID: "ec0c26de-cbe1-462a-9d0c-ada687f84a78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.382866 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.382759    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-podinfo\") pod \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\" (UID: \"ec0c26de-cbe1-462a-9d0c-ada687f84a78\") "
Nov 02 12:52:03.383307 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.382980    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-socket" pod "ec0c26de-cbe1-462a-9d0c-ada687f84a78" (UID: "ec0c26de-cbe1-462a-9d0c-ada687f84a78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.383441 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.383420    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.383542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.383451    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.383542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.383491    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.383542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.383519    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.383789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.383769    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istiod-ca-cert" pod "ec0c26de-cbe1-462a-9d0c-ada687f84a78" (UID: "ec0c26de-cbe1-462a-9d0c-ada687f84a78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:03.385278 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.385249    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-token" pod "ec0c26de-cbe1-462a-9d0c-ada687f84a78" (UID: "ec0c26de-cbe1-462a-9d0c-ada687f84a78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.385977 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.385953    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-envoy" pod "ec0c26de-cbe1-462a-9d0c-ada687f84a78" (UID: "ec0c26de-cbe1-462a-9d0c-ada687f84a78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.387072 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.386950    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-podinfo" pod "ec0c26de-cbe1-462a-9d0c-ada687f84a78" (UID: "ec0c26de-cbe1-462a-9d0c-ada687f84a78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:52:03.387228 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.387090    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-kube-api-access-lxpcc" pod "ec0c26de-cbe1-462a-9d0c-ada687f84a78" (UID: "ec0c26de-cbe1-462a-9d0c-ada687f84a78"). InnerVolumeSpecName "kube-api-access-lxpcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.401225 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.401188    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:52:03.484875 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.484839    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-data\") pod \"7036741f-e614-4dc6-b368-10e452e8310a\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") "
Nov 02 12:52:03.485105 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.484895    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-credential-socket\") pod \"7036741f-e614-4dc6-b368-10e452e8310a\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") "
Nov 02 12:52:03.485105 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.484969    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-istio-token\" (UniqueName: \"kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-istio-token\") pod \"7036741f-e614-4dc6-b368-10e452e8310a\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") "
Nov 02 12:52:03.485105 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.485099    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-kube-api-access-zpkrx\" (UniqueName: \"kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-kube-api-access-zpkrx\") pod \"7036741f-e614-4dc6-b368-10e452e8310a\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") "
Nov 02 12:52:03.485288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.485170    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-socket\") pod \"7036741f-e614-4dc6-b368-10e452e8310a\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") "
Nov 02 12:52:03.485288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.485177    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-credential-socket" pod "7036741f-e614-4dc6-b368-10e452e8310a" (UID: "7036741f-e614-4dc6-b368-10e452e8310a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.485288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.485184    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-data" pod "7036741f-e614-4dc6-b368-10e452e8310a" (UID: "7036741f-e614-4dc6-b368-10e452e8310a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.485288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.485269    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-certs\") pod \"7036741f-e614-4dc6-b368-10e452e8310a\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") "
Nov 02 12:52:03.485509 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.485370    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7036741f-e614-4dc6-b368-10e452e8310a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7036741f-e614-4dc6-b368-10e452e8310a-istio-podinfo\") pod \"7036741f-e614-4dc6-b368-10e452e8310a\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") "
Nov 02 12:52:03.485509 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.485403    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-socket" pod "7036741f-e614-4dc6-b368-10e452e8310a" (UID: "7036741f-e614-4dc6-b368-10e452e8310a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.485509 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.485443    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7036741f-e614-4dc6-b368-10e452e8310a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7036741f-e614-4dc6-b368-10e452e8310a-istiod-ca-cert\") pod \"7036741f-e614-4dc6-b368-10e452e8310a\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") "
Nov 02 12:52:03.485509 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.485498    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-envoy\") pod \"7036741f-e614-4dc6-b368-10e452e8310a\" (UID: \"7036741f-e614-4dc6-b368-10e452e8310a\") "
Nov 02 12:52:03.485733 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.485519    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-certs" pod "7036741f-e614-4dc6-b368-10e452e8310a" (UID: "7036741f-e614-4dc6-b368-10e452e8310a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.485879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.485855    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7036741f-e614-4dc6-b368-10e452e8310a-istiod-ca-cert" pod "7036741f-e614-4dc6-b368-10e452e8310a" (UID: "7036741f-e614-4dc6-b368-10e452e8310a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:03.486103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.486081    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.486202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.486109    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.486202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.486123    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.486202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.486137    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.486202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.486149    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.486202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.486162    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.486202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.486175    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec0c26de-cbe1-462a-9d0c-ada687f84a78-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.486202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.486187    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7036741f-e614-4dc6-b368-10e452e8310a-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.486202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.486199    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxpcc\" (UniqueName: \"kubernetes.io/projected/ec0c26de-cbe1-462a-9d0c-ada687f84a78-kube-api-access-lxpcc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.486788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.486213    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.487931 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.487901    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-kube-api-access-zpkrx" pod "7036741f-e614-4dc6-b368-10e452e8310a" (UID: "7036741f-e614-4dc6-b368-10e452e8310a"). InnerVolumeSpecName "kube-api-access-zpkrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.488097 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.488045    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7036741f-e614-4dc6-b368-10e452e8310a-istio-podinfo" pod "7036741f-e614-4dc6-b368-10e452e8310a" (UID: "7036741f-e614-4dc6-b368-10e452e8310a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:52:03.488203 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.488094    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-istio-token" pod "7036741f-e614-4dc6-b368-10e452e8310a" (UID: "7036741f-e614-4dc6-b368-10e452e8310a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:03.488281 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.488205    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-envoy" pod "7036741f-e614-4dc6-b368-10e452e8310a" (UID: "7036741f-e614-4dc6-b368-10e452e8310a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:52:03.496811 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.496791    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:03.587872 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.587833    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpkrx\" (UniqueName: \"kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-kube-api-access-zpkrx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.587872 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.587869    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7036741f-e614-4dc6-b368-10e452e8310a-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.587872 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.587882    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7036741f-e614-4dc6-b368-10e452e8310a-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.588146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.587890    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7036741f-e614-4dc6-b368-10e452e8310a-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:03.683726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.683652    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"]
Nov 02 12:52:03.684246 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684233    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f926c748-f9e9-4be9-9ae4-a06c61a76631" containerName="revoke-keys"
Nov 02 12:52:03.684300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684248    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f926c748-f9e9-4be9-9ae4-a06c61a76631" containerName="revoke-keys"
Nov 02 12:52:03.684300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684261    5879 cpu_manager.go:414] "containerMap: removing container" podUID="344cbaff-2812-4910-a6dc-7c33fd030d39" containerName="istio-proxy"
Nov 02 12:52:03.684300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684267    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="344cbaff-2812-4910-a6dc-7c33fd030d39" containerName="istio-proxy"
Nov 02 12:52:03.684300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684278    5879 cpu_manager.go:414] "containerMap: removing container" podUID="07b7068f-e026-4857-bac2-bd487fca458e" containerName="revoke-keys"
Nov 02 12:52:03.684300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684284    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b7068f-e026-4857-bac2-bd487fca458e" containerName="revoke-keys"
Nov 02 12:52:03.684300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684297    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ec0c26de-cbe1-462a-9d0c-ada687f84a78" containerName="istio-proxy"
Nov 02 12:52:03.684300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684302    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec0c26de-cbe1-462a-9d0c-ada687f84a78" containerName="istio-proxy"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684309    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1816f3b7-b324-45cd-add2-9931d7f1eb26" containerName="istio-proxy"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684314    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1816f3b7-b324-45cd-add2-9931d7f1eb26" containerName="istio-proxy"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684321    5879 cpu_manager.go:414] "containerMap: removing container" podUID="25b9b75f-b1e5-4e8c-9989-c4d3059422d2" containerName="revoke-keys"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684325    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b9b75f-b1e5-4e8c-9989-c4d3059422d2" containerName="revoke-keys"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684333    5879 cpu_manager.go:414] "containerMap: removing container" podUID="7036741f-e614-4dc6-b368-10e452e8310a" containerName="istio-proxy"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684337    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7036741f-e614-4dc6-b368-10e452e8310a" containerName="istio-proxy"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684349    5879 cpu_manager.go:414] "containerMap: removing container" podUID="143c9cce-2a0f-454d-b363-87ebf116e254" containerName="istio-proxy"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684353    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143c9cce-2a0f-454d-b363-87ebf116e254" containerName="istio-proxy"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684454    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="7036741f-e614-4dc6-b368-10e452e8310a" containerName="istio-proxy"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684462    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="143c9cce-2a0f-454d-b363-87ebf116e254" containerName="istio-proxy"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684474    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f926c748-f9e9-4be9-9ae4-a06c61a76631" containerName="revoke-keys"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684480    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="344cbaff-2812-4910-a6dc-7c33fd030d39" containerName="istio-proxy"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684487    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b7068f-e026-4857-bac2-bd487fca458e" containerName="revoke-keys"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684494    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec0c26de-cbe1-462a-9d0c-ada687f84a78" containerName="istio-proxy"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684500    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1816f3b7-b324-45cd-add2-9931d7f1eb26" containerName="istio-proxy"
Nov 02 12:52:03.684510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.684506    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b9b75f-b1e5-4e8c-9989-c4d3059422d2" containerName="revoke-keys"
Nov 02 12:52:03.689162 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.689139    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:03.709118 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.708675    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:52:03.709118 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.708808    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:52:03.709118 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.709049    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"]
Nov 02 12:52:03.791100 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.791070    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59eac350-dljsz\" (UID: \"583f2743-034a-4154-9c77-aed8d9fe3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:03.791266 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.791189    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njk6h\" (UniqueName: \"kubernetes.io/projected/583f2743-034a-4154-9c77-aed8d9fe3e6f-kube-api-access-njk6h\") pod \"maas-api-revoke-sub-59eac350-dljsz\" (UID: \"583f2743-034a-4154-9c77-aed8d9fe3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:03.791351 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.791334    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59eac350-dljsz\" (UID: \"583f2743-034a-4154-9c77-aed8d9fe3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:03.892798 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.892765    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59eac350-dljsz\" (UID: \"583f2743-034a-4154-9c77-aed8d9fe3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:03.893007 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.892865    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njk6h\" (UniqueName: \"kubernetes.io/projected/583f2743-034a-4154-9c77-aed8d9fe3e6f-kube-api-access-njk6h\") pod \"maas-api-revoke-sub-59eac350-dljsz\" (UID: \"583f2743-034a-4154-9c77-aed8d9fe3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:03.893076 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.893037    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59eac350-dljsz\" (UID: \"583f2743-034a-4154-9c77-aed8d9fe3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:03.893906 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.893880    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59eac350-dljsz\" (UID: \"583f2743-034a-4154-9c77-aed8d9fe3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:03.896105 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.896084    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59eac350-dljsz\" (UID: \"583f2743-034a-4154-9c77-aed8d9fe3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:03.923332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:03.923308    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njk6h\" (UniqueName: \"kubernetes.io/projected/583f2743-034a-4154-9c77-aed8d9fe3e6f-kube-api-access-njk6h\") pod \"maas-api-revoke-sub-59eac350-dljsz\" (UID: \"583f2743-034a-4154-9c77-aed8d9fe3e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:04.009681 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.009586    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:04.211481 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.211448    5879 generic.go:356] "Generic (PLEG): container finished" podID="7036741f-e614-4dc6-b368-10e452e8310a" containerID="ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659" exitCode=0
Nov 02 12:52:04.211938 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.211491    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"
Nov 02 12:52:04.211938 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.211537    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222" event={"ID":"7036741f-e614-4dc6-b368-10e452e8310a","Type":"ContainerDied","Data":"ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659"}
Nov 02 12:52:04.211938 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.211588    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222" event={"ID":"7036741f-e614-4dc6-b368-10e452e8310a","Type":"ContainerDied","Data":"2d1c274dd4529e53a16d251ccfc197c15bcb8e1bc21ca413660c4eb1957254c7"}
Nov 02 12:52:04.211938 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.211612    5879 scope.go:122] "RemoveContainer" containerID="ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659"
Nov 02 12:52:04.213211 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.213193    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"
Nov 02 12:52:04.213337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.213215    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv" event={"ID":"ec0c26de-cbe1-462a-9d0c-ada687f84a78","Type":"ContainerDied","Data":"b9c9ca06ac6a819c268c306e2a3184aa7a14aec4658dadfaa8c8c97ab16bd6ca"}
Nov 02 12:52:04.222093 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.222077    5879 scope.go:122] "RemoveContainer" containerID="ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659"
Nov 02 12:52:04.222385 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:52:04.222368    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659\": container with ID starting with ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659 not found: ID does not exist" containerID="ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659"
Nov 02 12:52:04.222433 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.222395    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659"} err="failed to get container status \"ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659\": rpc error: code = NotFound desc = could not find container \"ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659\": container with ID starting with ee5997b10a0f12bdac8c09cd3c2deb5051802356fb85d76b66503a1dae267659 not found: ID does not exist"
Nov 02 12:52:04.222433 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.222411    5879 scope.go:122] "RemoveContainer" containerID="8f145e506bd40b11daa786109cb081e838c7563fe81e0f12ddb213eab90dc6ac"
Nov 02 12:52:04.268668 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.268584    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"]
Nov 02 12:52:04.281537 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.281509    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-7s5dv"]
Nov 02 12:52:04.309735 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.309704    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"]
Nov 02 12:52:04.313774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.313745    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-9750c7-openshift-default-54d64b97d5-vp222"]
Nov 02 12:52:04.316783 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:04.316759    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"]
Nov 02 12:52:04.320307 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:52:04.320281    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583f2743_034a_4154_9c77_aed8d9fe3e6f.slice/crio-f3e7910c3bad2e370d3c5c555f161a2ec532eb5b417a09d76466e79e1868b40a WatchSource:0}: Error finding container f3e7910c3bad2e370d3c5c555f161a2ec532eb5b417a09d76466e79e1868b40a: Status 404 returned error can't find the container with id f3e7910c3bad2e370d3c5c555f161a2ec532eb5b417a09d76466e79e1868b40a
Nov 02 12:52:05.212272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.212241    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:05.222565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.222528    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz" event={"ID":"583f2743-034a-4154-9c77-aed8d9fe3e6f","Type":"ContainerStarted","Data":"a2b2480333e2fd070f2fdaf9ebdcf7be6777c762620373b5de92c161fcdd0f25"}
Nov 02 12:52:05.222750 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.222582    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz" event={"ID":"583f2743-034a-4154-9c77-aed8d9fe3e6f","Type":"ContainerStarted","Data":"f3e7910c3bad2e370d3c5c555f161a2ec532eb5b417a09d76466e79e1868b40a"}
Nov 02 12:52:05.266525 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.266469    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz" podStartSLOduration=2.266456806 podStartE2EDuration="2.266456806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:05.266039465 +0000 UTC m=+1471.372742317" watchObservedRunningTime="2026-11-02 12:52:05.266456806 +0000 UTC m=+1471.373159657"
Nov 02 12:52:05.438740 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.438702    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"]
Nov 02 12:52:05.443407 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.443375    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:05.463483 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.463413    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"]
Nov 02 12:52:05.616143 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.616111    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efe51eb8-hzr7r\" (UID: \"58527ce8-efa6-4415-a874-5d9676f27e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:05.616143 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.616145    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efe51eb8-hzr7r\" (UID: \"58527ce8-efa6-4415-a874-5d9676f27e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:05.616346 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.616288    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q77w\" (UniqueName: \"kubernetes.io/projected/58527ce8-efa6-4415-a874-5d9676f27e07-kube-api-access-9q77w\") pod \"maas-api-revoke-sub-efe51eb8-hzr7r\" (UID: \"58527ce8-efa6-4415-a874-5d9676f27e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:05.642302 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.642268    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:05.718140 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.718044    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q77w\" (UniqueName: \"kubernetes.io/projected/58527ce8-efa6-4415-a874-5d9676f27e07-kube-api-access-9q77w\") pod \"maas-api-revoke-sub-efe51eb8-hzr7r\" (UID: \"58527ce8-efa6-4415-a874-5d9676f27e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:05.718333 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.718315    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efe51eb8-hzr7r\" (UID: \"58527ce8-efa6-4415-a874-5d9676f27e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:05.718394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.718348    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efe51eb8-hzr7r\" (UID: \"58527ce8-efa6-4415-a874-5d9676f27e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:05.719079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.719033    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efe51eb8-hzr7r\" (UID: \"58527ce8-efa6-4415-a874-5d9676f27e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:05.721161 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.721132    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efe51eb8-hzr7r\" (UID: \"58527ce8-efa6-4415-a874-5d9676f27e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:05.762918 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:05.762881    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q77w\" (UniqueName: \"kubernetes.io/projected/58527ce8-efa6-4415-a874-5d9676f27e07-kube-api-access-9q77w\") pod \"maas-api-revoke-sub-efe51eb8-hzr7r\" (UID: \"58527ce8-efa6-4415-a874-5d9676f27e07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:06.017078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.016978    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7036741f-e614-4dc6-b368-10e452e8310a" path="/var/lib/kubelet/pods/7036741f-e614-4dc6-b368-10e452e8310a/volumes"
Nov 02 12:52:06.017685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.017669    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec0c26de-cbe1-462a-9d0c-ada687f84a78" path="/var/lib/kubelet/pods/ec0c26de-cbe1-462a-9d0c-ada687f84a78/volumes"
Nov 02 12:52:06.057968 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.057935    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:06.105090 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.105047    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:06.231604 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.231568    5879 generic.go:356] "Generic (PLEG): container finished" podID="583f2743-034a-4154-9c77-aed8d9fe3e6f" containerID="a2b2480333e2fd070f2fdaf9ebdcf7be6777c762620373b5de92c161fcdd0f25" exitCode=0
Nov 02 12:52:06.232074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.231661    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz" event={"ID":"583f2743-034a-4154-9c77-aed8d9fe3e6f","Type":"ContainerDied","Data":"a2b2480333e2fd070f2fdaf9ebdcf7be6777c762620373b5de92c161fcdd0f25"}
Nov 02 12:52:06.382133 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.382101    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"]
Nov 02 12:52:06.387729 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.387705    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:06.419487 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:52:06.419460    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58527ce8_efa6_4415_a874_5d9676f27e07.slice/crio-da4bf6b6ac55087fa0667be1af9b105363416d1bdbf37d13046b51e7240be9cf WatchSource:0}: Error finding container da4bf6b6ac55087fa0667be1af9b105363416d1bdbf37d13046b51e7240be9cf: Status 404 returned error can't find the container with id da4bf6b6ac55087fa0667be1af9b105363416d1bdbf37d13046b51e7240be9cf
Nov 02 12:52:06.420334 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.420306    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"]
Nov 02 12:52:06.473852 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.473820    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"]
Nov 02 12:52:06.528451 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.528417    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f182b01c-hdwtv\" (UID: \"90a126d6-15b8-4395-b94d-853bbee759b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:06.528636 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.528478    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9nv9\" (UniqueName: \"kubernetes.io/projected/90a126d6-15b8-4395-b94d-853bbee759b2-kube-api-access-j9nv9\") pod \"maas-api-revoke-sub-f182b01c-hdwtv\" (UID: \"90a126d6-15b8-4395-b94d-853bbee759b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:06.528693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.528673    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f182b01c-hdwtv\" (UID: \"90a126d6-15b8-4395-b94d-853bbee759b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:06.629693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.629657    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f182b01c-hdwtv\" (UID: \"90a126d6-15b8-4395-b94d-853bbee759b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:06.629876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.629780    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f182b01c-hdwtv\" (UID: \"90a126d6-15b8-4395-b94d-853bbee759b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:06.629876 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.629818    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9nv9\" (UniqueName: \"kubernetes.io/projected/90a126d6-15b8-4395-b94d-853bbee759b2-kube-api-access-j9nv9\") pod \"maas-api-revoke-sub-f182b01c-hdwtv\" (UID: \"90a126d6-15b8-4395-b94d-853bbee759b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:06.630456 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.630428    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f182b01c-hdwtv\" (UID: \"90a126d6-15b8-4395-b94d-853bbee759b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:06.632244 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.632195    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f182b01c-hdwtv\" (UID: \"90a126d6-15b8-4395-b94d-853bbee759b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:06.658401 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.657322    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9nv9\" (UniqueName: \"kubernetes.io/projected/90a126d6-15b8-4395-b94d-853bbee759b2-kube-api-access-j9nv9\") pod \"maas-api-revoke-sub-f182b01c-hdwtv\" (UID: \"90a126d6-15b8-4395-b94d-853bbee759b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:06.703765 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:06.703742    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:07.009977 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:52:07.009946    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a126d6_15b8_4395_b94d_853bbee759b2.slice/crio-c558e03c40ef858197cfd9bdb7d603118e684f12d3ff9f382105aa1ecb99d823 WatchSource:0}: Error finding container c558e03c40ef858197cfd9bdb7d603118e684f12d3ff9f382105aa1ecb99d823: Status 404 returned error can't find the container with id c558e03c40ef858197cfd9bdb7d603118e684f12d3ff9f382105aa1ecb99d823
Nov 02 12:52:07.009977 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.009960    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"]
Nov 02 12:52:07.239662 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.239629    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r" event={"ID":"58527ce8-efa6-4415-a874-5d9676f27e07","Type":"ContainerStarted","Data":"19138d1e0cfcf0ded571850e3e64c5cfdaac762d8ac8f4144ecc6f3174450d32"}
Nov 02 12:52:07.240085 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.239686    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r" event={"ID":"58527ce8-efa6-4415-a874-5d9676f27e07","Type":"ContainerStarted","Data":"da4bf6b6ac55087fa0667be1af9b105363416d1bdbf37d13046b51e7240be9cf"}
Nov 02 12:52:07.241130 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.241101    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv" event={"ID":"90a126d6-15b8-4395-b94d-853bbee759b2","Type":"ContainerStarted","Data":"c558e03c40ef858197cfd9bdb7d603118e684f12d3ff9f382105aa1ecb99d823"}
Nov 02 12:52:07.288275 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.288214    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r" podStartSLOduration=2.28819244 podStartE2EDuration="2.28819244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:07.284290016 +0000 UTC m=+1473.390992879" watchObservedRunningTime="2026-11-02 12:52:07.28819244 +0000 UTC m=+1473.394895296"
Nov 02 12:52:07.416047 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.415981    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:07.450033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.449975    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:07.540462 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.540370    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-openshift-ca\") pod \"583f2743-034a-4154-9c77-aed8d9fe3e6f\" (UID: \"583f2743-034a-4154-9c77-aed8d9fe3e6f\") "
Nov 02 12:52:07.540635 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.540545    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-xks-ca\") pod \"583f2743-034a-4154-9c77-aed8d9fe3e6f\" (UID: \"583f2743-034a-4154-9c77-aed8d9fe3e6f\") "
Nov 02 12:52:07.540813 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.540789    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-openshift-ca" pod "583f2743-034a-4154-9c77-aed8d9fe3e6f" (UID: "583f2743-034a-4154-9c77-aed8d9fe3e6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:07.541124 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.541102    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583f2743-034a-4154-9c77-aed8d9fe3e6f-kube-api-access-njk6h\" (UniqueName: \"kubernetes.io/projected/583f2743-034a-4154-9c77-aed8d9fe3e6f-kube-api-access-njk6h\") pod \"583f2743-034a-4154-9c77-aed8d9fe3e6f\" (UID: \"583f2743-034a-4154-9c77-aed8d9fe3e6f\") "
Nov 02 12:52:07.542477 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.542453    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:07.544673 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.544632    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-xks-ca" pod "583f2743-034a-4154-9c77-aed8d9fe3e6f" (UID: "583f2743-034a-4154-9c77-aed8d9fe3e6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:07.551160 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.551116    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583f2743-034a-4154-9c77-aed8d9fe3e6f-kube-api-access-njk6h" pod "583f2743-034a-4154-9c77-aed8d9fe3e6f" (UID: "583f2743-034a-4154-9c77-aed8d9fe3e6f"). InnerVolumeSpecName "kube-api-access-njk6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:07.634156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.634117    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:07.645833 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.645800    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njk6h\" (UniqueName: \"kubernetes.io/projected/583f2743-034a-4154-9c77-aed8d9fe3e6f-kube-api-access-njk6h\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:07.645833 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.645833    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/583f2743-034a-4154-9c77-aed8d9fe3e6f-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:07.782190 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:07.782146    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:08.251649 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:08.251557    5879 generic.go:356] "Generic (PLEG): container finished" podID="58527ce8-efa6-4415-a874-5d9676f27e07" containerID="19138d1e0cfcf0ded571850e3e64c5cfdaac762d8ac8f4144ecc6f3174450d32" exitCode=0
Nov 02 12:52:08.252304 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:08.251610    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r" event={"ID":"58527ce8-efa6-4415-a874-5d9676f27e07","Type":"ContainerDied","Data":"19138d1e0cfcf0ded571850e3e64c5cfdaac762d8ac8f4144ecc6f3174450d32"}
Nov 02 12:52:08.258375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:08.256362    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz" event={"ID":"583f2743-034a-4154-9c77-aed8d9fe3e6f","Type":"ContainerDied","Data":"f3e7910c3bad2e370d3c5c555f161a2ec532eb5b417a09d76466e79e1868b40a"}
Nov 02 12:52:08.258375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:08.256427    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3e7910c3bad2e370d3c5c555f161a2ec532eb5b417a09d76466e79e1868b40a"
Nov 02 12:52:08.258375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:08.256508    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"
Nov 02 12:52:08.261877 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:08.261838    5879 generic.go:356] "Generic (PLEG): container finished" podID="90a126d6-15b8-4395-b94d-853bbee759b2" containerID="48ce97e16de4723bf7d53dfcb15ab806e4ca0d5cff8dc9106cd587ce8c61b874" exitCode=0
Nov 02 12:52:08.262086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:08.261917    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv" event={"ID":"90a126d6-15b8-4395-b94d-853bbee759b2","Type":"ContainerDied","Data":"48ce97e16de4723bf7d53dfcb15ab806e4ca0d5cff8dc9106cd587ce8c61b874"}
Nov 02 12:52:08.269149 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:08.269122    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:09.432759 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.432729    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:09.504314 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.504279    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:09.581228 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.581129    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-openshift-ca\") pod \"58527ce8-efa6-4415-a874-5d9676f27e07\" (UID: \"58527ce8-efa6-4415-a874-5d9676f27e07\") "
Nov 02 12:52:09.581426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.581248    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58527ce8-efa6-4415-a874-5d9676f27e07-kube-api-access-9q77w\" (UniqueName: \"kubernetes.io/projected/58527ce8-efa6-4415-a874-5d9676f27e07-kube-api-access-9q77w\") pod \"58527ce8-efa6-4415-a874-5d9676f27e07\" (UID: \"58527ce8-efa6-4415-a874-5d9676f27e07\") "
Nov 02 12:52:09.581426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.581325    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-xks-ca\") pod \"58527ce8-efa6-4415-a874-5d9676f27e07\" (UID: \"58527ce8-efa6-4415-a874-5d9676f27e07\") "
Nov 02 12:52:09.581560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.581542    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-openshift-ca" pod "58527ce8-efa6-4415-a874-5d9676f27e07" (UID: "58527ce8-efa6-4415-a874-5d9676f27e07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:09.581909 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.581879    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:09.584605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.584567    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-xks-ca" pod "58527ce8-efa6-4415-a874-5d9676f27e07" (UID: "58527ce8-efa6-4415-a874-5d9676f27e07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:09.584605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.584575    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58527ce8-efa6-4415-a874-5d9676f27e07-kube-api-access-9q77w" pod "58527ce8-efa6-4415-a874-5d9676f27e07" (UID: "58527ce8-efa6-4415-a874-5d9676f27e07"). InnerVolumeSpecName "kube-api-access-9q77w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:09.683221 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.683189    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a126d6-15b8-4395-b94d-853bbee759b2-kube-api-access-j9nv9\" (UniqueName: \"kubernetes.io/projected/90a126d6-15b8-4395-b94d-853bbee759b2-kube-api-access-j9nv9\") pod \"90a126d6-15b8-4395-b94d-853bbee759b2\" (UID: \"90a126d6-15b8-4395-b94d-853bbee759b2\") "
Nov 02 12:52:09.683432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.683276    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-xks-ca\") pod \"90a126d6-15b8-4395-b94d-853bbee759b2\" (UID: \"90a126d6-15b8-4395-b94d-853bbee759b2\") "
Nov 02 12:52:09.683432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.683394    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-openshift-ca\") pod \"90a126d6-15b8-4395-b94d-853bbee759b2\" (UID: \"90a126d6-15b8-4395-b94d-853bbee759b2\") "
Nov 02 12:52:09.683758 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.683728    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-openshift-ca" pod "90a126d6-15b8-4395-b94d-853bbee759b2" (UID: "90a126d6-15b8-4395-b94d-853bbee759b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:09.683920 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.683907    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58527ce8-efa6-4415-a874-5d9676f27e07-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:09.683964 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.683922    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:09.683964 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.683934    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q77w\" (UniqueName: \"kubernetes.io/projected/58527ce8-efa6-4415-a874-5d9676f27e07-kube-api-access-9q77w\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:09.685542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.685519    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a126d6-15b8-4395-b94d-853bbee759b2-kube-api-access-j9nv9" pod "90a126d6-15b8-4395-b94d-853bbee759b2" (UID: "90a126d6-15b8-4395-b94d-853bbee759b2"). InnerVolumeSpecName "kube-api-access-j9nv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:09.686481 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.686455    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-xks-ca" pod "90a126d6-15b8-4395-b94d-853bbee759b2" (UID: "90a126d6-15b8-4395-b94d-853bbee759b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:09.785459 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.785429    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9nv9\" (UniqueName: \"kubernetes.io/projected/90a126d6-15b8-4395-b94d-853bbee759b2-kube-api-access-j9nv9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:09.785459 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.785458    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90a126d6-15b8-4395-b94d-853bbee759b2-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:09.870208 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:09.870179    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:10.276422 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:10.276336    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"
Nov 02 12:52:10.276422 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:10.276365    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv" event={"ID":"90a126d6-15b8-4395-b94d-853bbee759b2","Type":"ContainerDied","Data":"c558e03c40ef858197cfd9bdb7d603118e684f12d3ff9f382105aa1ecb99d823"}
Nov 02 12:52:10.276610 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:10.276443    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c558e03c40ef858197cfd9bdb7d603118e684f12d3ff9f382105aa1ecb99d823"
Nov 02 12:52:10.277767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:10.277745    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r" event={"ID":"58527ce8-efa6-4415-a874-5d9676f27e07","Type":"ContainerDied","Data":"da4bf6b6ac55087fa0667be1af9b105363416d1bdbf37d13046b51e7240be9cf"}
Nov 02 12:52:10.277881 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:10.277775    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"
Nov 02 12:52:10.277881 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:10.277785    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da4bf6b6ac55087fa0667be1af9b105363416d1bdbf37d13046b51e7240be9cf"
Nov 02 12:52:11.460945 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:11.460907    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:12.026494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:12.026463    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:12.226424 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:12.226374    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:12.722227 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:12.722192    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:13.083334 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:13.083243    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:13.380501 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:13.380463    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:14.829374 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:52:14.829336    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:52:15.169227 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:15.169188    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:16.683469 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.683434    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"]
Nov 02 12:52:16.684105 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.684089    5879 cpu_manager.go:414] "containerMap: removing container" podUID="58527ce8-efa6-4415-a874-5d9676f27e07" containerName="revoke-keys"
Nov 02 12:52:16.684105 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.684105    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58527ce8-efa6-4415-a874-5d9676f27e07" containerName="revoke-keys"
Nov 02 12:52:16.684223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.684159    5879 cpu_manager.go:414] "containerMap: removing container" podUID="90a126d6-15b8-4395-b94d-853bbee759b2" containerName="revoke-keys"
Nov 02 12:52:16.684223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.684165    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a126d6-15b8-4395-b94d-853bbee759b2" containerName="revoke-keys"
Nov 02 12:52:16.684223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.684175    5879 cpu_manager.go:414] "containerMap: removing container" podUID="583f2743-034a-4154-9c77-aed8d9fe3e6f" containerName="revoke-keys"
Nov 02 12:52:16.684223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.684180    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583f2743-034a-4154-9c77-aed8d9fe3e6f" containerName="revoke-keys"
Nov 02 12:52:16.684338 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.684282    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a126d6-15b8-4395-b94d-853bbee759b2" containerName="revoke-keys"
Nov 02 12:52:16.684338 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.684290    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="583f2743-034a-4154-9c77-aed8d9fe3e6f" containerName="revoke-keys"
Nov 02 12:52:16.684338 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.684302    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="58527ce8-efa6-4415-a874-5d9676f27e07" containerName="revoke-keys"
Nov 02 12:52:16.689330 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.689297    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:16.706856 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.706827    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"]
Nov 02 12:52:16.716963 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.716936    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:52:16.717146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.717038    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:52:16.746089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.746058    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcdb9655-b573-4310-82be-ca490663c074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d64ce5-n55hb\" (UID: \"fcdb9655-b573-4310-82be-ca490663c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:16.746285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.746112    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcdb9655-b573-4310-82be-ca490663c074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d64ce5-n55hb\" (UID: \"fcdb9655-b573-4310-82be-ca490663c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:16.746285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.746202    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gntd\" (UniqueName: \"kubernetes.io/projected/fcdb9655-b573-4310-82be-ca490663c074-kube-api-access-4gntd\") pod \"maas-api-revoke-sub-25d64ce5-n55hb\" (UID: \"fcdb9655-b573-4310-82be-ca490663c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:16.847528 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.847494    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcdb9655-b573-4310-82be-ca490663c074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d64ce5-n55hb\" (UID: \"fcdb9655-b573-4310-82be-ca490663c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:16.847722 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.847538    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcdb9655-b573-4310-82be-ca490663c074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d64ce5-n55hb\" (UID: \"fcdb9655-b573-4310-82be-ca490663c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:16.847722 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.847563    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gntd\" (UniqueName: \"kubernetes.io/projected/fcdb9655-b573-4310-82be-ca490663c074-kube-api-access-4gntd\") pod \"maas-api-revoke-sub-25d64ce5-n55hb\" (UID: \"fcdb9655-b573-4310-82be-ca490663c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:16.848182 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.848155    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcdb9655-b573-4310-82be-ca490663c074-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25d64ce5-n55hb\" (UID: \"fcdb9655-b573-4310-82be-ca490663c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:16.850244 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.850218    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcdb9655-b573-4310-82be-ca490663c074-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25d64ce5-n55hb\" (UID: \"fcdb9655-b573-4310-82be-ca490663c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:16.874263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:16.874238    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gntd\" (UniqueName: \"kubernetes.io/projected/fcdb9655-b573-4310-82be-ca490663c074-kube-api-access-4gntd\") pod \"maas-api-revoke-sub-25d64ce5-n55hb\" (UID: \"fcdb9655-b573-4310-82be-ca490663c074\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:17.004329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:17.004241    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:17.319284 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:52:17.319238    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdb9655_b573_4310_82be_ca490663c074.slice/crio-b9f3f2fb2ea49a54a80c27e51c0481c623d0348384f868619c46935f9005ec38 WatchSource:0}: Error finding container b9f3f2fb2ea49a54a80c27e51c0481c623d0348384f868619c46935f9005ec38: Status 404 returned error can't find the container with id b9f3f2fb2ea49a54a80c27e51c0481c623d0348384f868619c46935f9005ec38
Nov 02 12:52:17.320204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:17.320181    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"]
Nov 02 12:52:17.329727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:17.329701    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb" event={"ID":"fcdb9655-b573-4310-82be-ca490663c074","Type":"ContainerStarted","Data":"b9f3f2fb2ea49a54a80c27e51c0481c623d0348384f868619c46935f9005ec38"}
Nov 02 12:52:17.484660 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:17.484630    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:18.337564 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:18.337528    5879 generic.go:356] "Generic (PLEG): container finished" podID="fcdb9655-b573-4310-82be-ca490663c074" containerID="82a5e412a600048f1921c1a045898ccb0f1700871e8da570da2f63b1014aa0b6" exitCode=0
Nov 02 12:52:18.338063 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:18.337607    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb" event={"ID":"fcdb9655-b573-4310-82be-ca490663c074","Type":"ContainerDied","Data":"82a5e412a600048f1921c1a045898ccb0f1700871e8da570da2f63b1014aa0b6"}
Nov 02 12:52:18.912019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:18.904023    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:20.283842 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.266556    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 12:52:20.377715 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.374923    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:20.456020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.455253    5879 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:20.599364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.593607    5879 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:20.604044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.599530    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:20.685173 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.685133    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb" event={"ID":"fcdb9655-b573-4310-82be-ca490663c074","Type":"ContainerDied","Data":"b9f3f2fb2ea49a54a80c27e51c0481c623d0348384f868619c46935f9005ec38"}
Nov 02 12:52:20.687636 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.687611    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9f3f2fb2ea49a54a80c27e51c0481c623d0348384f868619c46935f9005ec38"
Nov 02 12:52:20.695103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.695062    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:20.840040 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.832062    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdb9655-b573-4310-82be-ca490663c074-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcdb9655-b573-4310-82be-ca490663c074-maas-api-openshift-ca\") pod \"fcdb9655-b573-4310-82be-ca490663c074\" (UID: \"fcdb9655-b573-4310-82be-ca490663c074\") "
Nov 02 12:52:20.840040 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.832209    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdb9655-b573-4310-82be-ca490663c074-kube-api-access-4gntd\" (UniqueName: \"kubernetes.io/projected/fcdb9655-b573-4310-82be-ca490663c074-kube-api-access-4gntd\") pod \"fcdb9655-b573-4310-82be-ca490663c074\" (UID: \"fcdb9655-b573-4310-82be-ca490663c074\") "
Nov 02 12:52:20.840040 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.832329    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcdb9655-b573-4310-82be-ca490663c074-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcdb9655-b573-4310-82be-ca490663c074-maas-api-xks-ca\") pod \"fcdb9655-b573-4310-82be-ca490663c074\" (UID: \"fcdb9655-b573-4310-82be-ca490663c074\") "
Nov 02 12:52:20.840040 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.834175    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdb9655-b573-4310-82be-ca490663c074-maas-api-openshift-ca" pod "fcdb9655-b573-4310-82be-ca490663c074" (UID: "fcdb9655-b573-4310-82be-ca490663c074"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:20.840040 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.837172    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdb9655-b573-4310-82be-ca490663c074-kube-api-access-4gntd" pod "fcdb9655-b573-4310-82be-ca490663c074" (UID: "fcdb9655-b573-4310-82be-ca490663c074"). InnerVolumeSpecName "kube-api-access-4gntd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:20.840040 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.839294    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdb9655-b573-4310-82be-ca490663c074-maas-api-xks-ca" pod "fcdb9655-b573-4310-82be-ca490663c074" (UID: "fcdb9655-b573-4310-82be-ca490663c074"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:20.955020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.936134    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcdb9655-b573-4310-82be-ca490663c074-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:20.955020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.936176    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gntd\" (UniqueName: \"kubernetes.io/projected/fcdb9655-b573-4310-82be-ca490663c074-kube-api-access-4gntd\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:20.955020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:20.936194    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcdb9655-b573-4310-82be-ca490663c074-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:21.641627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:21.641587    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:21.692133 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:21.692101    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"
Nov 02 12:52:22.208433 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.208397    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:22.414133 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.414099    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"]
Nov 02 12:52:22.414678 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.414665    5879 cpu_manager.go:414] "containerMap: removing container" podUID="fcdb9655-b573-4310-82be-ca490663c074" containerName="revoke-keys"
Nov 02 12:52:22.414742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.414681    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdb9655-b573-4310-82be-ca490663c074" containerName="revoke-keys"
Nov 02 12:52:22.414779 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.414769    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdb9655-b573-4310-82be-ca490663c074" containerName="revoke-keys"
Nov 02 12:52:22.418483 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.418459    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:22.434658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.434630    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:52:22.438893 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.438856    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"]
Nov 02 12:52:22.448404 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.448376    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:52:22.455435 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.455398    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603b62b8-kjh95\" (UID: \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:22.455571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.455454    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwnzs\" (UniqueName: \"kubernetes.io/projected/f2523a19-b10c-47f6-81cc-71a8082b2ae9-kube-api-access-gwnzs\") pod \"maas-api-revoke-sub-603b62b8-kjh95\" (UID: \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:22.455571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.455532    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603b62b8-kjh95\" (UID: \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:22.557481 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.557385    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603b62b8-kjh95\" (UID: \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:22.557481 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.557442    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwnzs\" (UniqueName: \"kubernetes.io/projected/f2523a19-b10c-47f6-81cc-71a8082b2ae9-kube-api-access-gwnzs\") pod \"maas-api-revoke-sub-603b62b8-kjh95\" (UID: \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:22.557700 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.557557    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603b62b8-kjh95\" (UID: \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:22.558223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.558200    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-603b62b8-kjh95\" (UID: \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:22.560120 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.560104    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-603b62b8-kjh95\" (UID: \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:22.582139 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.582107    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwnzs\" (UniqueName: \"kubernetes.io/projected/f2523a19-b10c-47f6-81cc-71a8082b2ae9-kube-api-access-gwnzs\") pod \"maas-api-revoke-sub-603b62b8-kjh95\" (UID: \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:22.631917 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.631880    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"]
Nov 02 12:52:22.635885 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.635865    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:22.656542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.656510    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"]
Nov 02 12:52:22.659330 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.659300    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2018b275-f8kmc\" (UID: \"63b06bdd-5931-4326-b010-f2b311cffe9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:22.659484 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.659442    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd968\" (UniqueName: \"kubernetes.io/projected/63b06bdd-5931-4326-b010-f2b311cffe9b-kube-api-access-nd968\") pod \"maas-api-revoke-sub-2018b275-f8kmc\" (UID: \"63b06bdd-5931-4326-b010-f2b311cffe9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:22.659560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.659535    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2018b275-f8kmc\" (UID: \"63b06bdd-5931-4326-b010-f2b311cffe9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:22.733236 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.733204    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:22.761263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.761231    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2018b275-f8kmc\" (UID: \"63b06bdd-5931-4326-b010-f2b311cffe9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:22.761460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.761348    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd968\" (UniqueName: \"kubernetes.io/projected/63b06bdd-5931-4326-b010-f2b311cffe9b-kube-api-access-nd968\") pod \"maas-api-revoke-sub-2018b275-f8kmc\" (UID: \"63b06bdd-5931-4326-b010-f2b311cffe9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:22.761529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.761471    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2018b275-f8kmc\" (UID: \"63b06bdd-5931-4326-b010-f2b311cffe9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:22.761987 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.761958    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2018b275-f8kmc\" (UID: \"63b06bdd-5931-4326-b010-f2b311cffe9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:22.764489 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.764462    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2018b275-f8kmc\" (UID: \"63b06bdd-5931-4326-b010-f2b311cffe9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:22.786325 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.786292    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd968\" (UniqueName: \"kubernetes.io/projected/63b06bdd-5931-4326-b010-f2b311cffe9b-kube-api-access-nd968\") pod \"maas-api-revoke-sub-2018b275-f8kmc\" (UID: \"63b06bdd-5931-4326-b010-f2b311cffe9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:22.950658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:22.950624    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:23.045464 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:52:23.045431    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2523a19_b10c_47f6_81cc_71a8082b2ae9.slice/crio-4b09a2d4afaced38068e723c295625c9a8705c4ea81da1070092524a83f66b08 WatchSource:0}: Error finding container 4b09a2d4afaced38068e723c295625c9a8705c4ea81da1070092524a83f66b08: Status 404 returned error can't find the container with id 4b09a2d4afaced38068e723c295625c9a8705c4ea81da1070092524a83f66b08
Nov 02 12:52:23.045780 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:23.045752    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"]
Nov 02 12:52:23.258694 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:52:23.258665    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b06bdd_5931_4326_b010_f2b311cffe9b.slice/crio-3a10ee38223123bef5d379529b424b2a9a8509e32316641b08e6c1acee560545 WatchSource:0}: Error finding container 3a10ee38223123bef5d379529b424b2a9a8509e32316641b08e6c1acee560545: Status 404 returned error can't find the container with id 3a10ee38223123bef5d379529b424b2a9a8509e32316641b08e6c1acee560545
Nov 02 12:52:23.261764 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:23.261742    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"]
Nov 02 12:52:23.708125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:23.708090    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc" event={"ID":"63b06bdd-5931-4326-b010-f2b311cffe9b","Type":"ContainerStarted","Data":"3e345b9f21ff5ccab9ea2813a7d1f161194067897cedf06e9f01444c75108028"}
Nov 02 12:52:23.708639 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:23.708192    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc" event={"ID":"63b06bdd-5931-4326-b010-f2b311cffe9b","Type":"ContainerStarted","Data":"3a10ee38223123bef5d379529b424b2a9a8509e32316641b08e6c1acee560545"}
Nov 02 12:52:23.709637 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:23.709612    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95" event={"ID":"f2523a19-b10c-47f6-81cc-71a8082b2ae9","Type":"ContainerStarted","Data":"b61840a347f356b073db776648dd954f03843930de6cf3ba54adf4b0ac1f7455"}
Nov 02 12:52:23.709756 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:23.709657    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95" event={"ID":"f2523a19-b10c-47f6-81cc-71a8082b2ae9","Type":"ContainerStarted","Data":"4b09a2d4afaced38068e723c295625c9a8705c4ea81da1070092524a83f66b08"}
Nov 02 12:52:23.753707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:23.753636    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc" podStartSLOduration=1.753616383 podStartE2EDuration="1.753616383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:23.752142857 +0000 UTC m=+1489.858845708" watchObservedRunningTime="2026-11-02 12:52:23.753616383 +0000 UTC m=+1489.860319236"
Nov 02 12:52:23.798762 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:23.798707    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95" podStartSLOduration=1.7986935050000001 podStartE2EDuration="1.798693505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:23.795317894 +0000 UTC m=+1489.902020745" watchObservedRunningTime="2026-11-02 12:52:23.798693505 +0000 UTC m=+1489.905396355"
Nov 02 12:52:24.165218 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:24.165176    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:24.718710 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:24.718672    5879 generic.go:356] "Generic (PLEG): container finished" podID="63b06bdd-5931-4326-b010-f2b311cffe9b" containerID="3e345b9f21ff5ccab9ea2813a7d1f161194067897cedf06e9f01444c75108028" exitCode=0
Nov 02 12:52:24.719149 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:24.718724    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc" event={"ID":"63b06bdd-5931-4326-b010-f2b311cffe9b","Type":"ContainerDied","Data":"3e345b9f21ff5ccab9ea2813a7d1f161194067897cedf06e9f01444c75108028"}
Nov 02 12:52:24.720276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:24.720247    5879 generic.go:356] "Generic (PLEG): container finished" podID="f2523a19-b10c-47f6-81cc-71a8082b2ae9" containerID="b61840a347f356b073db776648dd954f03843930de6cf3ba54adf4b0ac1f7455" exitCode=0
Nov 02 12:52:24.720371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:24.720322    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95" event={"ID":"f2523a19-b10c-47f6-81cc-71a8082b2ae9","Type":"ContainerDied","Data":"b61840a347f356b073db776648dd954f03843930de6cf3ba54adf4b0ac1f7455"}
Nov 02 12:52:25.902588 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:25.902553    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:25.940696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:25.940666    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:25.945207 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:25.945180    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:26.019607 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.019525    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-openshift-ca\") pod \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\" (UID: \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\") "
Nov 02 12:52:26.019758 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.019605    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-xks-ca\") pod \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\" (UID: \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\") "
Nov 02 12:52:26.019758 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.019681    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-openshift-ca\") pod \"63b06bdd-5931-4326-b010-f2b311cffe9b\" (UID: \"63b06bdd-5931-4326-b010-f2b311cffe9b\") "
Nov 02 12:52:26.019758 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.019723    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-xks-ca\") pod \"63b06bdd-5931-4326-b010-f2b311cffe9b\" (UID: \"63b06bdd-5931-4326-b010-f2b311cffe9b\") "
Nov 02 12:52:26.019939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.019779    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b06bdd-5931-4326-b010-f2b311cffe9b-kube-api-access-nd968\" (UniqueName: \"kubernetes.io/projected/63b06bdd-5931-4326-b010-f2b311cffe9b-kube-api-access-nd968\") pod \"63b06bdd-5931-4326-b010-f2b311cffe9b\" (UID: \"63b06bdd-5931-4326-b010-f2b311cffe9b\") "
Nov 02 12:52:26.019939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.019925    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2523a19-b10c-47f6-81cc-71a8082b2ae9-kube-api-access-gwnzs\" (UniqueName: \"kubernetes.io/projected/f2523a19-b10c-47f6-81cc-71a8082b2ae9-kube-api-access-gwnzs\") pod \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\" (UID: \"f2523a19-b10c-47f6-81cc-71a8082b2ae9\") "
Nov 02 12:52:26.020154 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.020130    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-openshift-ca" pod "63b06bdd-5931-4326-b010-f2b311cffe9b" (UID: "63b06bdd-5931-4326-b010-f2b311cffe9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:26.020406 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.020354    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-openshift-ca" pod "f2523a19-b10c-47f6-81cc-71a8082b2ae9" (UID: "f2523a19-b10c-47f6-81cc-71a8082b2ae9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:26.020580 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.020530    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:26.020580 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.020550    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:26.022907 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.022876    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-xks-ca" pod "f2523a19-b10c-47f6-81cc-71a8082b2ae9" (UID: "f2523a19-b10c-47f6-81cc-71a8082b2ae9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:26.023066 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.023038    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b06bdd-5931-4326-b010-f2b311cffe9b-kube-api-access-nd968" pod "63b06bdd-5931-4326-b010-f2b311cffe9b" (UID: "63b06bdd-5931-4326-b010-f2b311cffe9b"). InnerVolumeSpecName "kube-api-access-nd968". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:26.023141 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.023082    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2523a19-b10c-47f6-81cc-71a8082b2ae9-kube-api-access-gwnzs" pod "f2523a19-b10c-47f6-81cc-71a8082b2ae9" (UID: "f2523a19-b10c-47f6-81cc-71a8082b2ae9"). InnerVolumeSpecName "kube-api-access-gwnzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:26.023620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.023572    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-xks-ca" pod "63b06bdd-5931-4326-b010-f2b311cffe9b" (UID: "63b06bdd-5931-4326-b010-f2b311cffe9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:26.033232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.033186    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:26.121685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.121655    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b06bdd-5931-4326-b010-f2b311cffe9b-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:26.121685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.121681    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd968\" (UniqueName: \"kubernetes.io/projected/63b06bdd-5931-4326-b010-f2b311cffe9b-kube-api-access-nd968\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:26.121685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.121704    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwnzs\" (UniqueName: \"kubernetes.io/projected/f2523a19-b10c-47f6-81cc-71a8082b2ae9-kube-api-access-gwnzs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:26.121948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.121713    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2523a19-b10c-47f6-81cc-71a8082b2ae9-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:26.735357 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.735325    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"
Nov 02 12:52:26.735357 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.735343    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc" event={"ID":"63b06bdd-5931-4326-b010-f2b311cffe9b","Type":"ContainerDied","Data":"3a10ee38223123bef5d379529b424b2a9a8509e32316641b08e6c1acee560545"}
Nov 02 12:52:26.735577 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.735394    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a10ee38223123bef5d379529b424b2a9a8509e32316641b08e6c1acee560545"
Nov 02 12:52:26.736804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.736780    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95" event={"ID":"f2523a19-b10c-47f6-81cc-71a8082b2ae9","Type":"ContainerDied","Data":"4b09a2d4afaced38068e723c295625c9a8705c4ea81da1070092524a83f66b08"}
Nov 02 12:52:26.736804 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.736797    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"
Nov 02 12:52:26.736973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:26.736821    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b09a2d4afaced38068e723c295625c9a8705c4ea81da1070092524a83f66b08"
Nov 02 12:52:27.752202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:27.752162    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:28.011743 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:28.011628    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:29.393410 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:29.393354    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:31.022683 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:31.022647    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:32.566052 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:32.565957    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:33.038029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.037974    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"]
Nov 02 12:52:33.038655 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.038640    5879 cpu_manager.go:414] "containerMap: removing container" podUID="63b06bdd-5931-4326-b010-f2b311cffe9b" containerName="revoke-keys"
Nov 02 12:52:33.038726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.038657    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b06bdd-5931-4326-b010-f2b311cffe9b" containerName="revoke-keys"
Nov 02 12:52:33.038726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.038674    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f2523a19-b10c-47f6-81cc-71a8082b2ae9" containerName="revoke-keys"
Nov 02 12:52:33.038726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.038682    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2523a19-b10c-47f6-81cc-71a8082b2ae9" containerName="revoke-keys"
Nov 02 12:52:33.038834 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.038771    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b06bdd-5931-4326-b010-f2b311cffe9b" containerName="revoke-keys"
Nov 02 12:52:33.038834 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.038786    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2523a19-b10c-47f6-81cc-71a8082b2ae9" containerName="revoke-keys"
Nov 02 12:52:33.044148 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.044119    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.069949 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.069917    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"]
Nov 02 12:52:33.224564 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.224521    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8276q\" (UniqueName: \"kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-kube-api-access-8276q\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.224732 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.224579    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.224795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.224746    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.224871 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.224851    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.224970 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.224889    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.224970 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.224928    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.225093 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.225023    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752a2b49-f48e-4c28-8595-8ed0a708b928-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.225093 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.225079    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.225162 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.225144    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.326613 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.326512    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.326613 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.326555    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.326613 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.326595    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.326913 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.326633    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752a2b49-f48e-4c28-8595-8ed0a708b928-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.326913 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.326695    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.326913 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.326857    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.327089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.327015    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.327089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.326985    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.327089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.327037    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8276q\" (UniqueName: \"kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-kube-api-access-8276q\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.327089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.327087    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.327293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.327172    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.327293 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.327181    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.327433 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.327409    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.327553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.327447    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752a2b49-f48e-4c28-8595-8ed0a708b928-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.329373 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.329344    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.329583 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.329565    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.352095 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.352066    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.356930 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.356899    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8276q\" (UniqueName: \"kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-kube-api-access-8276q\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.375032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.374980    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:33.699564 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:52:33.699534    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752a2b49_f48e_4c28_8595_8ed0a708b928.slice/crio-6b307064f69d1175f26de1bed1a493027531c3f8d0c0a4e6ebeea423bb61f3d2 WatchSource:0}: Error finding container 6b307064f69d1175f26de1bed1a493027531c3f8d0c0a4e6ebeea423bb61f3d2: Status 404 returned error can't find the container with id 6b307064f69d1175f26de1bed1a493027531c3f8d0c0a4e6ebeea423bb61f3d2
Nov 02 12:52:33.700464 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.700440    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"]
Nov 02 12:52:33.701774 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.701742    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:52:33.701852 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.701805    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:52:33.701852 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.701836    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:52:33.794908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.794872    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" event={"ID":"752a2b49-f48e-4c28-8595-8ed0a708b928","Type":"ContainerStarted","Data":"673e0159cf29ef458c209aedd01897298a6cfa0f0986d64c7ee257f83577016c"}
Nov 02 12:52:33.795133 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.794930    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" event={"ID":"752a2b49-f48e-4c28-8595-8ed0a708b928","Type":"ContainerStarted","Data":"6b307064f69d1175f26de1bed1a493027531c3f8d0c0a4e6ebeea423bb61f3d2"}
Nov 02 12:52:33.846339 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:33.846284    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" podStartSLOduration=0.846268253 podStartE2EDuration="846.268253ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:33.842719906 +0000 UTC m=+1499.949422759" watchObservedRunningTime="2026-11-02 12:52:33.846268253 +0000 UTC m=+1499.952971103"
Nov 02 12:52:34.375495 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:34.375459    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:34.377636 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:34.377609    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:52:34.377890 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:34.377855    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 12:52:34.466304 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:34.466260    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:34.520685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:34.520654    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"]
Nov 02 12:52:34.520918 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:34.520893    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f" podUID="46418b5e-3bb7-4f4a-b5e0-86dba55616d5" containerName="payload-processing" containerID="cri-o://c49c17df416c32d5d649c88d21edc6c58557d5951612b2c2479212fd3a78eec4" gracePeriod=30
Nov 02 12:52:34.659086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:34.658229    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:34.826727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:34.820022    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:34.858400 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:34.855699    5879 generic.go:356] "Generic (PLEG): container finished" podID="46418b5e-3bb7-4f4a-b5e0-86dba55616d5" containerID="c49c17df416c32d5d649c88d21edc6c58557d5951612b2c2479212fd3a78eec4" exitCode=0
Nov 02 12:52:34.858400 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:34.856518    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f" event={"ID":"46418b5e-3bb7-4f4a-b5e0-86dba55616d5","Type":"ContainerDied","Data":"c49c17df416c32d5d649c88d21edc6c58557d5951612b2c2479212fd3a78eec4"}
Nov 02 12:52:35.019752 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.019721    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"
Nov 02 12:52:35.163162 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.163142    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-plugins-config-volume\") pod \"46418b5e-3bb7-4f4a-b5e0-86dba55616d5\" (UID: \"46418b5e-3bb7-4f4a-b5e0-86dba55616d5\") "
Nov 02 12:52:35.163326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.163311    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-kube-api-access-2wwxs\" (UniqueName: \"kubernetes.io/projected/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-kube-api-access-2wwxs\") pod \"46418b5e-3bb7-4f4a-b5e0-86dba55616d5\" (UID: \"46418b5e-3bb7-4f4a-b5e0-86dba55616d5\") "
Nov 02 12:52:35.163491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.163465    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-plugins-config-volume" pod "46418b5e-3bb7-4f4a-b5e0-86dba55616d5" (UID: "46418b5e-3bb7-4f4a-b5e0-86dba55616d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:35.164135 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.164109    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:35.165563 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.165544    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-kube-api-access-2wwxs" pod "46418b5e-3bb7-4f4a-b5e0-86dba55616d5" (UID: "46418b5e-3bb7-4f4a-b5e0-86dba55616d5"). InnerVolumeSpecName "kube-api-access-2wwxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:35.265374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.265296    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wwxs\" (UniqueName: \"kubernetes.io/projected/46418b5e-3bb7-4f4a-b5e0-86dba55616d5-kube-api-access-2wwxs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:35.312017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.311930    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:35.375548 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.375514    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:52:35.375743 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.375575    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 12:52:35.864517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.864487    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"
Nov 02 12:52:35.864517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.864509    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f" event={"ID":"46418b5e-3bb7-4f4a-b5e0-86dba55616d5","Type":"ContainerDied","Data":"b0e56a77d54ec4c41cfd1349788cd9a55cfe82bec41be547e87463291eb7159a"}
Nov 02 12:52:35.864970 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.864569    5879 scope.go:122] "RemoveContainer" containerID="c49c17df416c32d5d649c88d21edc6c58557d5951612b2c2479212fd3a78eec4"
Nov 02 12:52:35.931409 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.931359    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"]
Nov 02 12:52:35.944896 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:35.944869    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eb24df-978647c64-qz72f"]
Nov 02 12:52:36.016958 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.016928    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46418b5e-3bb7-4f4a-b5e0-86dba55616d5" path="/var/lib/kubelet/pods/46418b5e-3bb7-4f4a-b5e0-86dba55616d5/volumes"
Nov 02 12:52:36.207858 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.207756    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:36.375500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.375451    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:52:36.375707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.375524    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 12:52:36.714708 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.711490    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"]
Nov 02 12:52:36.714708 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.712357    5879 cpu_manager.go:414] "containerMap: removing container" podUID="46418b5e-3bb7-4f4a-b5e0-86dba55616d5" containerName="payload-processing"
Nov 02 12:52:36.714708 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.712377    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46418b5e-3bb7-4f4a-b5e0-86dba55616d5" containerName="payload-processing"
Nov 02 12:52:36.714708 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.712535    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="46418b5e-3bb7-4f4a-b5e0-86dba55616d5" containerName="payload-processing"
Nov 02 12:52:36.738512 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.733336    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"]
Nov 02 12:52:36.738512 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.733502    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:36.753035 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.752987    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:52:36.756934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.756904    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:52:36.795019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.792494    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:36.803017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.800918    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlbwr\" (UniqueName: \"kubernetes.io/projected/08e746bd-3082-4aad-8fb1-960b4b8c66a0-kube-api-access-rlbwr\") pod \"maas-api-revoke-sub-0114666b-2c28g\" (UID: \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:36.803017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.801101    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0114666b-2c28g\" (UID: \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:36.803017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.801171    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0114666b-2c28g\" (UID: \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:36.904143 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.904104    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlbwr\" (UniqueName: \"kubernetes.io/projected/08e746bd-3082-4aad-8fb1-960b4b8c66a0-kube-api-access-rlbwr\") pod \"maas-api-revoke-sub-0114666b-2c28g\" (UID: \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:36.904584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.904294    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0114666b-2c28g\" (UID: \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:36.904584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.904376    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0114666b-2c28g\" (UID: \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:36.905869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.905843    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0114666b-2c28g\" (UID: \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:36.974959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:36.973500    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0114666b-2c28g\" (UID: \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:37.032544 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:37.032506    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:37.037294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:37.037254    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlbwr\" (UniqueName: \"kubernetes.io/projected/08e746bd-3082-4aad-8fb1-960b4b8c66a0-kube-api-access-rlbwr\") pod \"maas-api-revoke-sub-0114666b-2c28g\" (UID: \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:37.059906 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:37.059862    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:37.191058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:37.182343    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:37.375469 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:37.375437    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:52:37.375658 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:37.375496    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 12:52:37.467191 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:52:37.467158    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e746bd_3082_4aad_8fb1_960b4b8c66a0.slice/crio-60ff651569e9d04e8cdafc487d375b7a523f0e7d9e9fa13aca58cfb0cac2b076 WatchSource:0}: Error finding container 60ff651569e9d04e8cdafc487d375b7a523f0e7d9e9fa13aca58cfb0cac2b076: Status 404 returned error can't find the container with id 60ff651569e9d04e8cdafc487d375b7a523f0e7d9e9fa13aca58cfb0cac2b076
Nov 02 12:52:37.468041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:37.468016    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"]
Nov 02 12:52:37.675242 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:37.675211    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:37.914909 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:37.914878    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g" event={"ID":"08e746bd-3082-4aad-8fb1-960b4b8c66a0","Type":"ContainerStarted","Data":"60ff651569e9d04e8cdafc487d375b7a523f0e7d9e9fa13aca58cfb0cac2b076"}
Nov 02 12:52:38.376529 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:38.376496    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:52:38.376721 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:38.376555    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 12:52:38.926947 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:38.926899    5879 generic.go:356] "Generic (PLEG): container finished" podID="08e746bd-3082-4aad-8fb1-960b4b8c66a0" containerID="c988e1dcbe85312560ffcf135423cf26c46e6badff3aeae987cdaed85ebbc0b9" exitCode=0
Nov 02 12:52:38.927401 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:38.927163    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g" event={"ID":"08e746bd-3082-4aad-8fb1-960b4b8c66a0","Type":"ContainerDied","Data":"c988e1dcbe85312560ffcf135423cf26c46e6badff3aeae987cdaed85ebbc0b9"}
Nov 02 12:52:39.376537 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:39.376109    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:52:39.376537 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:39.376172    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 12:52:39.418131 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:39.418098    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:40.696074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:40.693887    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:52:40.696074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:40.693944    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 12:52:41.044953 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.044852    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g" event={"ID":"08e746bd-3082-4aad-8fb1-960b4b8c66a0","Type":"ContainerDied","Data":"60ff651569e9d04e8cdafc487d375b7a523f0e7d9e9fa13aca58cfb0cac2b076"}
Nov 02 12:52:41.045280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.045235    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60ff651569e9d04e8cdafc487d375b7a523f0e7d9e9fa13aca58cfb0cac2b076"
Nov 02 12:52:41.051325 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.051302    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:41.148150 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.148117    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-xks-ca\") pod \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\" (UID: \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\") "
Nov 02 12:52:41.148567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.148546    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-openshift-ca\") pod \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\" (UID: \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\") "
Nov 02 12:52:41.148814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.148795    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08e746bd-3082-4aad-8fb1-960b4b8c66a0-kube-api-access-rlbwr\" (UniqueName: \"kubernetes.io/projected/08e746bd-3082-4aad-8fb1-960b4b8c66a0-kube-api-access-rlbwr\") pod \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\" (UID: \"08e746bd-3082-4aad-8fb1-960b4b8c66a0\") "
Nov 02 12:52:41.150744 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.150690    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-openshift-ca" pod "08e746bd-3082-4aad-8fb1-960b4b8c66a0" (UID: "08e746bd-3082-4aad-8fb1-960b4b8c66a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:41.157556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.157525    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:41.283472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.264493    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-xks-ca" pod "08e746bd-3082-4aad-8fb1-960b4b8c66a0" (UID: "08e746bd-3082-4aad-8fb1-960b4b8c66a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:41.283472 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.264820    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08e746bd-3082-4aad-8fb1-960b4b8c66a0-kube-api-access-rlbwr" pod "08e746bd-3082-4aad-8fb1-960b4b8c66a0" (UID: "08e746bd-3082-4aad-8fb1-960b4b8c66a0"). InnerVolumeSpecName "kube-api-access-rlbwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:41.286234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.286204    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08e746bd-3082-4aad-8fb1-960b4b8c66a0-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:41.286458 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.286430    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlbwr\" (UniqueName: \"kubernetes.io/projected/08e746bd-3082-4aad-8fb1-960b4b8c66a0-kube-api-access-rlbwr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:41.407022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.392945    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 02 12:52:41.427347 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.427302    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 02 12:52:41.427960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.409484    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:41.576016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.575962    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:41.956391 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:41.956346    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 12:52:42.048076 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:42.047428    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"
Nov 02 12:52:42.093514 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:42.093486    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:42.257193 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:42.257105    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:42.717687 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:42.717658    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:42.718066 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:42.718035    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:42.718930 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:42.718910    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:52:42.781230 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:42.781192    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:43.255702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.255672    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"]
Nov 02 12:52:43.256247 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.256233    5879 cpu_manager.go:414] "containerMap: removing container" podUID="08e746bd-3082-4aad-8fb1-960b4b8c66a0" containerName="revoke-keys"
Nov 02 12:52:43.256247 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.256247    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08e746bd-3082-4aad-8fb1-960b4b8c66a0" containerName="revoke-keys"
Nov 02 12:52:43.256435 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.256402    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="08e746bd-3082-4aad-8fb1-960b4b8c66a0" containerName="revoke-keys"
Nov 02 12:52:43.260173 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.260156    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:43.278627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.278600    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:52:43.278793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.278726    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:52:43.283847 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.283816    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"]
Nov 02 12:52:43.299588 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.299553    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada1da4c-zc6tv\" (UID: \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:43.299760 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.299711    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxfch\" (UniqueName: \"kubernetes.io/projected/f2b2b22e-5109-46e7-9cfe-2ef93976715d-kube-api-access-mxfch\") pod \"maas-api-revoke-sub-ada1da4c-zc6tv\" (UID: \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:43.299942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.299919    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada1da4c-zc6tv\" (UID: \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:43.401476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.401432    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada1da4c-zc6tv\" (UID: \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:43.401686 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.401606    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada1da4c-zc6tv\" (UID: \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:43.401768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.401709    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxfch\" (UniqueName: \"kubernetes.io/projected/f2b2b22e-5109-46e7-9cfe-2ef93976715d-kube-api-access-mxfch\") pod \"maas-api-revoke-sub-ada1da4c-zc6tv\" (UID: \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:43.402348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.402319    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada1da4c-zc6tv\" (UID: \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:43.404647 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.404624    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada1da4c-zc6tv\" (UID: \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:43.426793 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.426763    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxfch\" (UniqueName: \"kubernetes.io/projected/f2b2b22e-5109-46e7-9cfe-2ef93976715d-kube-api-access-mxfch\") pod \"maas-api-revoke-sub-ada1da4c-zc6tv\" (UID: \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:43.576878 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.576805    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:43.903288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:43.903248    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"]
Nov 02 12:52:44.061041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:44.060973    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv" event={"ID":"f2b2b22e-5109-46e7-9cfe-2ef93976715d","Type":"ContainerStarted","Data":"739d793678570b1defe0a7082490d4bea2d031bc64866266e2e90e95824eefe0"}
Nov 02 12:52:44.098959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:44.098927    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:44.454707 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:44.454622    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:44.591248 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:44.591210    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:44.835979 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:52:44.835899    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:52:45.068929 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:45.068890    5879 generic.go:356] "Generic (PLEG): container finished" podID="f2b2b22e-5109-46e7-9cfe-2ef93976715d" containerID="66c6f281dbb2dc0a83d0f56b49df28034f2adafe770fc471fe42843e005bc7f5" exitCode=0
Nov 02 12:52:45.069140 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:45.068978    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv" event={"ID":"f2b2b22e-5109-46e7-9cfe-2ef93976715d","Type":"ContainerDied","Data":"66c6f281dbb2dc0a83d0f56b49df28034f2adafe770fc471fe42843e005bc7f5"}
Nov 02 12:52:45.252468 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:45.252438    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:45.610662 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:45.610571    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:45.997960 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:45.997922    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:46.267430 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:46.267361    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:46.349112 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:46.349057    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-xks-ca\") pod \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\" (UID: \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\") "
Nov 02 12:52:46.349397 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:46.349134    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2b2b22e-5109-46e7-9cfe-2ef93976715d-kube-api-access-mxfch\" (UniqueName: \"kubernetes.io/projected/f2b2b22e-5109-46e7-9cfe-2ef93976715d-kube-api-access-mxfch\") pod \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\" (UID: \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\") "
Nov 02 12:52:46.349397 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:46.349316    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-openshift-ca\") pod \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\" (UID: \"f2b2b22e-5109-46e7-9cfe-2ef93976715d\") "
Nov 02 12:52:46.350268 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:46.350142    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-openshift-ca" pod "f2b2b22e-5109-46e7-9cfe-2ef93976715d" (UID: "f2b2b22e-5109-46e7-9cfe-2ef93976715d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:46.352263 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:46.352224    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2b2b22e-5109-46e7-9cfe-2ef93976715d-kube-api-access-mxfch" pod "f2b2b22e-5109-46e7-9cfe-2ef93976715d" (UID: "f2b2b22e-5109-46e7-9cfe-2ef93976715d"). InnerVolumeSpecName "kube-api-access-mxfch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:46.353435 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:46.353412    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-xks-ca" pod "f2b2b22e-5109-46e7-9cfe-2ef93976715d" (UID: "f2b2b22e-5109-46e7-9cfe-2ef93976715d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:46.435569 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:46.435535    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:46.451403 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:46.451358    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:46.451403 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:46.451396    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2b2b22e-5109-46e7-9cfe-2ef93976715d-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:46.451403 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:46.451406    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxfch\" (UniqueName: \"kubernetes.io/projected/f2b2b22e-5109-46e7-9cfe-2ef93976715d-kube-api-access-mxfch\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:47.082900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:47.082861    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"
Nov 02 12:52:47.082900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:47.082880    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv" event={"ID":"f2b2b22e-5109-46e7-9cfe-2ef93976715d","Type":"ContainerDied","Data":"739d793678570b1defe0a7082490d4bea2d031bc64866266e2e90e95824eefe0"}
Nov 02 12:52:47.083343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:47.082925    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="739d793678570b1defe0a7082490d4bea2d031bc64866266e2e90e95824eefe0"
Nov 02 12:52:47.805408 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:47.805368    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:50.057075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.057039    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:50.150479 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.150446    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"]
Nov 02 12:52:50.150982 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.150969    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f2b2b22e-5109-46e7-9cfe-2ef93976715d" containerName="revoke-keys"
Nov 02 12:52:50.151058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.150983    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2b2b22e-5109-46e7-9cfe-2ef93976715d" containerName="revoke-keys"
Nov 02 12:52:50.151132 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.151122    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2b2b22e-5109-46e7-9cfe-2ef93976715d" containerName="revoke-keys"
Nov 02 12:52:50.154893 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.154872    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:50.174068 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.174042    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"]
Nov 02 12:52:50.183814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.183785    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:52:50.184029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.183941    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:52:50.305241 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.305205    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1193ef8f-wh42v\" (UID: \"2eab034c-f96c-4890-a455-164c4e27bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:50.305241 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.305246    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1193ef8f-wh42v\" (UID: \"2eab034c-f96c-4890-a455-164c4e27bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:50.305476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.305442    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhw7l\" (UniqueName: \"kubernetes.io/projected/2eab034c-f96c-4890-a455-164c4e27bad7-kube-api-access-mhw7l\") pod \"maas-api-revoke-sub-1193ef8f-wh42v\" (UID: \"2eab034c-f96c-4890-a455-164c4e27bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:50.407173 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.407139    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhw7l\" (UniqueName: \"kubernetes.io/projected/2eab034c-f96c-4890-a455-164c4e27bad7-kube-api-access-mhw7l\") pod \"maas-api-revoke-sub-1193ef8f-wh42v\" (UID: \"2eab034c-f96c-4890-a455-164c4e27bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:50.407361 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.407293    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1193ef8f-wh42v\" (UID: \"2eab034c-f96c-4890-a455-164c4e27bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:50.407361 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.407324    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1193ef8f-wh42v\" (UID: \"2eab034c-f96c-4890-a455-164c4e27bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:50.408140 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.408113    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1193ef8f-wh42v\" (UID: \"2eab034c-f96c-4890-a455-164c4e27bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:50.410041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.409986    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1193ef8f-wh42v\" (UID: \"2eab034c-f96c-4890-a455-164c4e27bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:50.432218 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.432190    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhw7l\" (UniqueName: \"kubernetes.io/projected/2eab034c-f96c-4890-a455-164c4e27bad7-kube-api-access-mhw7l\") pod \"maas-api-revoke-sub-1193ef8f-wh42v\" (UID: \"2eab034c-f96c-4890-a455-164c4e27bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:50.469598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.469563    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:50.771938 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:52:50.771908    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eab034c_f96c_4890_a455_164c4e27bad7.slice/crio-6d65db35f08bda329c8a381f35f3c8655c52b787b156ff98a1d6f8588ef59f17 WatchSource:0}: Error finding container 6d65db35f08bda329c8a381f35f3c8655c52b787b156ff98a1d6f8588ef59f17: Status 404 returned error can't find the container with id 6d65db35f08bda329c8a381f35f3c8655c52b787b156ff98a1d6f8588ef59f17
Nov 02 12:52:50.772664 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:50.772644    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"]
Nov 02 12:52:51.122051 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:51.122014    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v" event={"ID":"2eab034c-f96c-4890-a455-164c4e27bad7","Type":"ContainerStarted","Data":"1aab9d948597b872ceea3aac15956a80188aef9afa70313cd029de67eb77aa9d"}
Nov 02 12:52:51.122451 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:51.122112    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v" event={"ID":"2eab034c-f96c-4890-a455-164c4e27bad7","Type":"ContainerStarted","Data":"6d65db35f08bda329c8a381f35f3c8655c52b787b156ff98a1d6f8588ef59f17"}
Nov 02 12:52:51.173916 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:51.173862    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v" podStartSLOduration=1.173847175 podStartE2EDuration="1.173847175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:51.172401877 +0000 UTC m=+1517.279104728" watchObservedRunningTime="2026-11-02 12:52:51.173847175 +0000 UTC m=+1517.280550028"
Nov 02 12:52:51.661022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:51.660972    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:52.055272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.055188    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"]
Nov 02 12:52:52.059483 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.059460    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.095948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.095915    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"]
Nov 02 12:52:52.131437 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.131404    5879 generic.go:356] "Generic (PLEG): container finished" podID="2eab034c-f96c-4890-a455-164c4e27bad7" containerID="1aab9d948597b872ceea3aac15956a80188aef9afa70313cd029de67eb77aa9d" exitCode=0
Nov 02 12:52:52.131825 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.131495    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v" event={"ID":"2eab034c-f96c-4890-a455-164c4e27bad7","Type":"ContainerDied","Data":"1aab9d948597b872ceea3aac15956a80188aef9afa70313cd029de67eb77aa9d"}
Nov 02 12:52:52.136119 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.136085    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b691630-bc7c-4785-939f-7eeb342f5264-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.136291 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.136234    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.136291 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.136260    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.136425 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.136396    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b691630-bc7c-4785-939f-7eeb342f5264-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.136509 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.136490    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.136566 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.136555    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.136747 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.136731    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.136807 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.136765    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.136900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.136879    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkn6t\" (UniqueName: \"kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-kube-api-access-hkn6t\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.238759 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.238722    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.238912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.238862    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.238912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.238887    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.238912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.238907    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkn6t\" (UniqueName: \"kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-kube-api-access-hkn6t\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.239070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.238974    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b691630-bc7c-4785-939f-7eeb342f5264-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.239123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.239071    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.239123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.238972    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.239123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.239107    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.239276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.239242    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.239276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.239248    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b691630-bc7c-4785-939f-7eeb342f5264-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.239375 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.239295    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.239556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.239533    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.239821 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.239718    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.240239 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.240215    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b691630-bc7c-4785-939f-7eeb342f5264-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.241560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.241535    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.241800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.241781    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b691630-bc7c-4785-939f-7eeb342f5264-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.265373 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.265341    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.265567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.265548    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkn6t\" (UniqueName: \"kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-kube-api-access-hkn6t\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.373056 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.372988    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:52.714267 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.714235    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"]
Nov 02 12:52:52.714475 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:52:52.714451    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b691630_bc7c_4785_939f_7eeb342f5264.slice/crio-2a61f51142ee66344348c8a9a385e5ee8749ba412020d463a77b9f540720c93d WatchSource:0}: Error finding container 2a61f51142ee66344348c8a9a385e5ee8749ba412020d463a77b9f540720c93d: Status 404 returned error can't find the container with id 2a61f51142ee66344348c8a9a385e5ee8749ba412020d463a77b9f540720c93d
Nov 02 12:52:52.717520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.717478    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:52:52.717633 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.717551    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:52:52.717633 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:52.717583    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:52:53.142384 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.142343    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" event={"ID":"3b691630-bc7c-4785-939f-7eeb342f5264","Type":"ContainerStarted","Data":"f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc"}
Nov 02 12:52:53.142821 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.142403    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" event={"ID":"3b691630-bc7c-4785-939f-7eeb342f5264","Type":"ContainerStarted","Data":"2a61f51142ee66344348c8a9a385e5ee8749ba412020d463a77b9f540720c93d"}
Nov 02 12:52:53.207049 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.206262    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" podStartSLOduration=1.206241281 podStartE2EDuration="1.206241281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:52:53.203981651 +0000 UTC m=+1519.310684521" watchObservedRunningTime="2026-11-02 12:52:53.206241281 +0000 UTC m=+1519.312944133"
Nov 02 12:52:53.299984 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.299956    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:53.378687 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.378637    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:52:53.390014 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.382417    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:53.390014 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.382478    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 12:52:53.460191 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.460107    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eab034c-f96c-4890-a455-164c4e27bad7-kube-api-access-mhw7l\" (UniqueName: \"kubernetes.io/projected/2eab034c-f96c-4890-a455-164c4e27bad7-kube-api-access-mhw7l\") pod \"2eab034c-f96c-4890-a455-164c4e27bad7\" (UID: \"2eab034c-f96c-4890-a455-164c4e27bad7\") "
Nov 02 12:52:53.460371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.460221    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-openshift-ca\") pod \"2eab034c-f96c-4890-a455-164c4e27bad7\" (UID: \"2eab034c-f96c-4890-a455-164c4e27bad7\") "
Nov 02 12:52:53.460371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.460367    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-xks-ca\") pod \"2eab034c-f96c-4890-a455-164c4e27bad7\" (UID: \"2eab034c-f96c-4890-a455-164c4e27bad7\") "
Nov 02 12:52:53.460626 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.460597    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-openshift-ca" pod "2eab034c-f96c-4890-a455-164c4e27bad7" (UID: "2eab034c-f96c-4890-a455-164c4e27bad7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:52:53.461120 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.460916    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:53.462809 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.462787    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eab034c-f96c-4890-a455-164c4e27bad7-kube-api-access-mhw7l" pod "2eab034c-f96c-4890-a455-164c4e27bad7" (UID: "2eab034c-f96c-4890-a455-164c4e27bad7"). InnerVolumeSpecName "kube-api-access-mhw7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:52:53.463705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.463679    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-xks-ca" pod "2eab034c-f96c-4890-a455-164c4e27bad7" (UID: "2eab034c-f96c-4890-a455-164c4e27bad7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:52:53.562767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.562733    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2eab034c-f96c-4890-a455-164c4e27bad7-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:53.562767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.562768    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhw7l\" (UniqueName: \"kubernetes.io/projected/2eab034c-f96c-4890-a455-164c4e27bad7-kube-api-access-mhw7l\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:52:53.563467 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.563443    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:53.774838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:53.774754    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:54.171018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:54.167573    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"
Nov 02 12:52:54.171018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:54.168248    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v" event={"ID":"2eab034c-f96c-4890-a455-164c4e27bad7","Type":"ContainerDied","Data":"6d65db35f08bda329c8a381f35f3c8655c52b787b156ff98a1d6f8588ef59f17"}
Nov 02 12:52:54.171018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:54.168293    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d65db35f08bda329c8a381f35f3c8655c52b787b156ff98a1d6f8588ef59f17"
Nov 02 12:52:54.375030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:54.373865    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:54.375030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:54.373918    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 12:52:55.377019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:55.376106    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:55.377019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:55.376171    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 12:52:55.827072 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:55.826426    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:55.902022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:55.901512    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 12:52:55.947077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:55.907454    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:55.947077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:55.932021    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.135344 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.134140    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.293959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.293920    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 12:52:56.327457 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.327419    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"]
Nov 02 12:52:56.328596 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.328568    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2eab034c-f96c-4890-a455-164c4e27bad7" containerName="revoke-keys"
Nov 02 12:52:56.341082 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.341044    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eab034c-f96c-4890-a455-164c4e27bad7" containerName="revoke-keys"
Nov 02 12:52:56.341613 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.341585    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eab034c-f96c-4890-a455-164c4e27bad7" containerName="revoke-keys"
Nov 02 12:52:56.363020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.351756    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:52:56.390097 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.385722    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:52:56.390097 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.386481    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.390097 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.387800    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:56.390097 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.387838    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 12:52:56.417044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.401220    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmkcw\" (UniqueName: \"kubernetes.io/projected/2a60794c-3304-4ea0-9a34-06e86b1be8eb-kube-api-access-wmkcw\") pod \"maas-api-revoke-sub-54b69fec-nqlxh\" (UID: \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:52:56.417044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.401345    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b69fec-nqlxh\" (UID: \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:52:56.417044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.405254    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b69fec-nqlxh\" (UID: \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:52:56.417364 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.417057    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:52:56.430574 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.430534    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"]
Nov 02 12:52:56.539021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.508615    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmkcw\" (UniqueName: \"kubernetes.io/projected/2a60794c-3304-4ea0-9a34-06e86b1be8eb-kube-api-access-wmkcw\") pod \"maas-api-revoke-sub-54b69fec-nqlxh\" (UID: \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:52:56.539490 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.539405    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b69fec-nqlxh\" (UID: \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:52:56.540066 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.539925    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b69fec-nqlxh\" (UID: \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:52:56.563566 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.563526    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54b69fec-nqlxh\" (UID: \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:52:56.768076 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.740720    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54b69fec-nqlxh\" (UID: \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:52:56.769801 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.769577    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmkcw\" (UniqueName: \"kubernetes.io/projected/2a60794c-3304-4ea0-9a34-06e86b1be8eb-kube-api-access-wmkcw\") pod \"maas-api-revoke-sub-54b69fec-nqlxh\" (UID: \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:52:56.929025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.902431    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:56.929025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.904240    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:52:56.937017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.929422    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 12:52:56.937017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.929839    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 12:52:56.937017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.930601    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:52:56.937017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.931672    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 12:52:56.937017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.931803    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:52:56.937017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.933142    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 12:52:56.937017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:56.933538    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:52:57.185019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.175102    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:57.213155 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.213116    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:52:57.493083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.340409    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 12:52:57.493083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.341074    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:52:57.493083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.464437    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:57.493083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.480124    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:52:57.588506 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.588464    5879 patch_prober.go:29] interesting pod/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 12:52:57.590038 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.588778    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:57.660499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.659610    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:57.660499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.659688    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 12:52:57.906121 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.906076    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 12:52:57.931017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.929554    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 12:52:57.975145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.931782    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:57.975145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.932102    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:52:57.975145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:57.934916    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 12:52:58.182016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.179081    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:58.190666 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.190634    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:52:58.198968 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.192368    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:52:58.422296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.372239    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 12:52:58.422296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.372314    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 12:52:58.422296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.378486    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 12:52:58.422296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.386218    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:58.422296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.386280    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 12:52:58.443047 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.442435    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:52:58.457754 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.456429    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:52:58.473779 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.473749    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:52:58.551667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.551639    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"]
Nov 02 12:52:58.559660 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.559613    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:52:58.841901 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:58.841858    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh" event={"ID":"2a60794c-3304-4ea0-9a34-06e86b1be8eb","Type":"ContainerStarted","Data":"55de41c0e7224aff301b620518542398e06d9c93a208accc015ffe67e36e981e"}
Nov 02 12:52:59.373457 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:59.373419    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:52:59.373656 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:59.373480    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 12:52:59.851222 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:59.851194    5879 generic.go:356] "Generic (PLEG): container finished" podID="2a60794c-3304-4ea0-9a34-06e86b1be8eb" containerID="10075cd7ba35d13fabe36d48bc122c2081486fea7aa326c17674d1e037d68468" exitCode=0
Nov 02 12:52:59.851606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:52:59.851279    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh" event={"ID":"2a60794c-3304-4ea0-9a34-06e86b1be8eb","Type":"ContainerDied","Data":"10075cd7ba35d13fabe36d48bc122c2081486fea7aa326c17674d1e037d68468"}
Nov 02 12:53:00.373799 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:00.373762    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:53:00.373964 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:00.373820    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 12:53:00.374940 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:00.374915    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:01.019947 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.019919    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:53:01.085065 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.085028    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"]
Nov 02 12:53:01.085936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.085907    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2a60794c-3304-4ea0-9a34-06e86b1be8eb" containerName="revoke-keys"
Nov 02 12:53:01.085936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.085930    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a60794c-3304-4ea0-9a34-06e86b1be8eb" containerName="revoke-keys"
Nov 02 12:53:01.086189 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.086084    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a60794c-3304-4ea0-9a34-06e86b1be8eb" containerName="revoke-keys"
Nov 02 12:53:01.089813 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.089795    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:01.109887 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.109823    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"]
Nov 02 12:53:01.125537 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.125502    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a60794c-3304-4ea0-9a34-06e86b1be8eb-kube-api-access-wmkcw\" (UniqueName: \"kubernetes.io/projected/2a60794c-3304-4ea0-9a34-06e86b1be8eb-kube-api-access-wmkcw\") pod \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\" (UID: \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\") "
Nov 02 12:53:01.125702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.125558    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-openshift-ca\") pod \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\" (UID: \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\") "
Nov 02 12:53:01.125702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.125596    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-xks-ca\") pod \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\" (UID: \"2a60794c-3304-4ea0-9a34-06e86b1be8eb\") "
Nov 02 12:53:01.126075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.126047    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-openshift-ca" pod "2a60794c-3304-4ea0-9a34-06e86b1be8eb" (UID: "2a60794c-3304-4ea0-9a34-06e86b1be8eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:53:01.127038 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.127011    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:01.128439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.128409    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-xks-ca" pod "2a60794c-3304-4ea0-9a34-06e86b1be8eb" (UID: "2a60794c-3304-4ea0-9a34-06e86b1be8eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:53:01.136595 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.136567    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a60794c-3304-4ea0-9a34-06e86b1be8eb-kube-api-access-wmkcw" pod "2a60794c-3304-4ea0-9a34-06e86b1be8eb" (UID: "2a60794c-3304-4ea0-9a34-06e86b1be8eb"). InnerVolumeSpecName "kube-api-access-wmkcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:01.228358 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.228325    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf104234-fe37-408f-9abd-9454197750a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d4cfbb3-6pgtg\" (UID: \"cf104234-fe37-408f-9abd-9454197750a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:01.228524 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.228384    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwf8\" (UniqueName: \"kubernetes.io/projected/cf104234-fe37-408f-9abd-9454197750a8-kube-api-access-dfwf8\") pod \"maas-api-revoke-sub-4d4cfbb3-6pgtg\" (UID: \"cf104234-fe37-408f-9abd-9454197750a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:01.228636 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.228610    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf104234-fe37-408f-9abd-9454197750a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d4cfbb3-6pgtg\" (UID: \"cf104234-fe37-408f-9abd-9454197750a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:01.228795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.228778    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmkcw\" (UniqueName: \"kubernetes.io/projected/2a60794c-3304-4ea0-9a34-06e86b1be8eb-kube-api-access-wmkcw\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:01.228881 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.228800    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a60794c-3304-4ea0-9a34-06e86b1be8eb-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:01.330301 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.330271    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf104234-fe37-408f-9abd-9454197750a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d4cfbb3-6pgtg\" (UID: \"cf104234-fe37-408f-9abd-9454197750a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:01.330476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.330459    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf104234-fe37-408f-9abd-9454197750a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d4cfbb3-6pgtg\" (UID: \"cf104234-fe37-408f-9abd-9454197750a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:01.330560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.330525    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwf8\" (UniqueName: \"kubernetes.io/projected/cf104234-fe37-408f-9abd-9454197750a8-kube-api-access-dfwf8\") pod \"maas-api-revoke-sub-4d4cfbb3-6pgtg\" (UID: \"cf104234-fe37-408f-9abd-9454197750a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:01.331149 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.331127    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf104234-fe37-408f-9abd-9454197750a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d4cfbb3-6pgtg\" (UID: \"cf104234-fe37-408f-9abd-9454197750a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:01.333084 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.333062    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf104234-fe37-408f-9abd-9454197750a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d4cfbb3-6pgtg\" (UID: \"cf104234-fe37-408f-9abd-9454197750a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:01.355435 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.355408    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwf8\" (UniqueName: \"kubernetes.io/projected/cf104234-fe37-408f-9abd-9454197750a8-kube-api-access-dfwf8\") pod \"maas-api-revoke-sub-4d4cfbb3-6pgtg\" (UID: \"cf104234-fe37-408f-9abd-9454197750a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:01.373813 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.373785    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 02 12:53:01.373959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.373842    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 02 12:53:01.415668 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.415637    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:01.723830 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:53:01.723803    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf104234_fe37_408f_9abd_9454197750a8.slice/crio-3aabfb2dd3133e5cc705239e13645724a906b5f34180943de1759ff59b8736ce WatchSource:0}: Error finding container 3aabfb2dd3133e5cc705239e13645724a906b5f34180943de1759ff59b8736ce: Status 404 returned error can't find the container with id 3aabfb2dd3133e5cc705239e13645724a906b5f34180943de1759ff59b8736ce
Nov 02 12:53:01.726140 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.726116    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"]
Nov 02 12:53:01.866751 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.866713    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:01.869442 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.869411    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg" event={"ID":"cf104234-fe37-408f-9abd-9454197750a8","Type":"ContainerStarted","Data":"3aabfb2dd3133e5cc705239e13645724a906b5f34180943de1759ff59b8736ce"}
Nov 02 12:53:01.871060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.871035    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"
Nov 02 12:53:01.871204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.871118    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh" event={"ID":"2a60794c-3304-4ea0-9a34-06e86b1be8eb","Type":"ContainerDied","Data":"55de41c0e7224aff301b620518542398e06d9c93a208accc015ffe67e36e981e"}
Nov 02 12:53:01.871204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:01.871168    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55de41c0e7224aff301b620518542398e06d9c93a208accc015ffe67e36e981e"
Nov 02 12:53:02.040217 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:02.040169    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:02.176904 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:02.176867    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:02.705564 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:02.705515    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:02.879530 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:02.879493    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg" event={"ID":"cf104234-fe37-408f-9abd-9454197750a8","Type":"ContainerStarted","Data":"0f543e5bd60c1fef3e7979c0ab6854e0d2412615216aa697e09c91dd638c3964"}
Nov 02 12:53:02.927109 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:02.927042    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg" podStartSLOduration=1.927026519 podStartE2EDuration="1.927026519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:02.925326686 +0000 UTC m=+1529.032029540" watchObservedRunningTime="2026-11-02 12:53:02.927026519 +0000 UTC m=+1529.033729370"
Nov 02 12:53:03.291270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:03.291237    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:53:03.291773 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:03.291691    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:53:03.292857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:03.292826    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:53:03.888302 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:03.888265    5879 generic.go:356] "Generic (PLEG): container finished" podID="cf104234-fe37-408f-9abd-9454197750a8" containerID="0f543e5bd60c1fef3e7979c0ab6854e0d2412615216aa697e09c91dd638c3964" exitCode=0
Nov 02 12:53:03.888497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:03.888351    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg" event={"ID":"cf104234-fe37-408f-9abd-9454197750a8","Type":"ContainerDied","Data":"0f543e5bd60c1fef3e7979c0ab6854e0d2412615216aa697e09c91dd638c3964"}
Nov 02 12:53:04.531983 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:04.531946    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:05.056984 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.056957    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:05.083456 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.083426    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf104234-fe37-408f-9abd-9454197750a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf104234-fe37-408f-9abd-9454197750a8-maas-api-xks-ca\") pod \"cf104234-fe37-408f-9abd-9454197750a8\" (UID: \"cf104234-fe37-408f-9abd-9454197750a8\") "
Nov 02 12:53:05.083620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.083550    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf104234-fe37-408f-9abd-9454197750a8-kube-api-access-dfwf8\" (UniqueName: \"kubernetes.io/projected/cf104234-fe37-408f-9abd-9454197750a8-kube-api-access-dfwf8\") pod \"cf104234-fe37-408f-9abd-9454197750a8\" (UID: \"cf104234-fe37-408f-9abd-9454197750a8\") "
Nov 02 12:53:05.083813 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.083795    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf104234-fe37-408f-9abd-9454197750a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf104234-fe37-408f-9abd-9454197750a8-maas-api-openshift-ca\") pod \"cf104234-fe37-408f-9abd-9454197750a8\" (UID: \"cf104234-fe37-408f-9abd-9454197750a8\") "
Nov 02 12:53:05.084219 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.084140    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf104234-fe37-408f-9abd-9454197750a8-maas-api-openshift-ca" pod "cf104234-fe37-408f-9abd-9454197750a8" (UID: "cf104234-fe37-408f-9abd-9454197750a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:53:05.084401 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.084356    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf104234-fe37-408f-9abd-9454197750a8-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:05.085811 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.085791    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf104234-fe37-408f-9abd-9454197750a8-kube-api-access-dfwf8" pod "cf104234-fe37-408f-9abd-9454197750a8" (UID: "cf104234-fe37-408f-9abd-9454197750a8"). InnerVolumeSpecName "kube-api-access-dfwf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:05.086041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.085988    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf104234-fe37-408f-9abd-9454197750a8-maas-api-xks-ca" pod "cf104234-fe37-408f-9abd-9454197750a8" (UID: "cf104234-fe37-408f-9abd-9454197750a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:53:05.185450 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.185367    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf104234-fe37-408f-9abd-9454197750a8-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:05.185450 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.185399    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfwf8\" (UniqueName: \"kubernetes.io/projected/cf104234-fe37-408f-9abd-9454197750a8-kube-api-access-dfwf8\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:05.903332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.903298    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"
Nov 02 12:53:05.903754 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.903350    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg" event={"ID":"cf104234-fe37-408f-9abd-9454197750a8","Type":"ContainerDied","Data":"3aabfb2dd3133e5cc705239e13645724a906b5f34180943de1759ff59b8736ce"}
Nov 02 12:53:05.903754 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:05.903450    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3aabfb2dd3133e5cc705239e13645724a906b5f34180943de1759ff59b8736ce"
Nov 02 12:53:06.244043 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:06.243950    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:07.951367 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:07.951338    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:08.745168 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.745132    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"]
Nov 02 12:53:08.745703 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.745689    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cf104234-fe37-408f-9abd-9454197750a8" containerName="revoke-keys"
Nov 02 12:53:08.745752 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.745706    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf104234-fe37-408f-9abd-9454197750a8" containerName="revoke-keys"
Nov 02 12:53:08.745834 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.745825    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf104234-fe37-408f-9abd-9454197750a8" containerName="revoke-keys"
Nov 02 12:53:08.749640 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.749625    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:08.769565 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.769539    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"]
Nov 02 12:53:08.778446 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.778422    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:53:08.778604 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.778499    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:53:08.830836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.830799    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be3f37f0-k6tt7\" (UID: \"959aa0ec-aca6-4538-85b9-f7ee508469fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:08.831081 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.831065    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdn4w\" (UniqueName: \"kubernetes.io/projected/959aa0ec-aca6-4538-85b9-f7ee508469fe-kube-api-access-wdn4w\") pod \"maas-api-revoke-sub-be3f37f0-k6tt7\" (UID: \"959aa0ec-aca6-4538-85b9-f7ee508469fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:08.831142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.831131    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be3f37f0-k6tt7\" (UID: \"959aa0ec-aca6-4538-85b9-f7ee508469fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:08.932965 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.932934    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdn4w\" (UniqueName: \"kubernetes.io/projected/959aa0ec-aca6-4538-85b9-f7ee508469fe-kube-api-access-wdn4w\") pod \"maas-api-revoke-sub-be3f37f0-k6tt7\" (UID: \"959aa0ec-aca6-4538-85b9-f7ee508469fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:08.933192 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.933021    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be3f37f0-k6tt7\" (UID: \"959aa0ec-aca6-4538-85b9-f7ee508469fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:08.933192 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.933056    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be3f37f0-k6tt7\" (UID: \"959aa0ec-aca6-4538-85b9-f7ee508469fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:08.933862 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.933793    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be3f37f0-k6tt7\" (UID: \"959aa0ec-aca6-4538-85b9-f7ee508469fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:08.935564 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.935541    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be3f37f0-k6tt7\" (UID: \"959aa0ec-aca6-4538-85b9-f7ee508469fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:08.958524 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:08.958494    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdn4w\" (UniqueName: \"kubernetes.io/projected/959aa0ec-aca6-4538-85b9-f7ee508469fe-kube-api-access-wdn4w\") pod \"maas-api-revoke-sub-be3f37f0-k6tt7\" (UID: \"959aa0ec-aca6-4538-85b9-f7ee508469fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:09.064052 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:09.063961    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:09.370962 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:53:09.370922    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959aa0ec_aca6_4538_85b9_f7ee508469fe.slice/crio-e8634be43e775191958582772926965c611332c8635dbd653dbf3172f3d92e6e WatchSource:0}: Error finding container e8634be43e775191958582772926965c611332c8635dbd653dbf3172f3d92e6e: Status 404 returned error can't find the container with id e8634be43e775191958582772926965c611332c8635dbd653dbf3172f3d92e6e
Nov 02 12:53:09.372158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:09.372136    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"]
Nov 02 12:53:09.557705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:09.557669    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:09.838800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:09.838764    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:09.933861 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:09.933821    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7" event={"ID":"959aa0ec-aca6-4538-85b9-f7ee508469fe","Type":"ContainerStarted","Data":"63f06613677a7d2efdcd9819bf7bffb68b6408b647660e95e32013ed95bf664a"}
Nov 02 12:53:09.934058 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:09.933888    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7" event={"ID":"959aa0ec-aca6-4538-85b9-f7ee508469fe","Type":"ContainerStarted","Data":"e8634be43e775191958582772926965c611332c8635dbd653dbf3172f3d92e6e"}
Nov 02 12:53:09.981584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:09.981530    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7" podStartSLOduration=1.981514862 podStartE2EDuration="1.981514862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:09.978985855 +0000 UTC m=+1536.085688707" watchObservedRunningTime="2026-11-02 12:53:09.981514862 +0000 UTC m=+1536.088217713"
Nov 02 12:53:10.035785 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:10.035753    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:10.616077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:10.611163    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:10.951881 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:10.948758    5879 generic.go:356] "Generic (PLEG): container finished" podID="959aa0ec-aca6-4538-85b9-f7ee508469fe" containerID="63f06613677a7d2efdcd9819bf7bffb68b6408b647660e95e32013ed95bf664a" exitCode=0
Nov 02 12:53:10.951881 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:10.948898    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7" event={"ID":"959aa0ec-aca6-4538-85b9-f7ee508469fe","Type":"ContainerDied","Data":"63f06613677a7d2efdcd9819bf7bffb68b6408b647660e95e32013ed95bf664a"}
Nov 02 12:53:11.008221 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:11.008182    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:11.724843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:11.724811    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:11.931606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:11.931574    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"]
Nov 02 12:53:11.935904 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:11.935885    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:11.962465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:11.962437    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"]
Nov 02 12:53:12.016728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.016632    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9b21fdf-7mln6\" (UID: \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:12.016728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.016709    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltp7t\" (UniqueName: \"kubernetes.io/projected/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-kube-api-access-ltp7t\") pod \"maas-api-revoke-sub-c9b21fdf-7mln6\" (UID: \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:12.017239 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.016894    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9b21fdf-7mln6\" (UID: \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:12.118604 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.118575    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9b21fdf-7mln6\" (UID: \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:12.118752 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.118726    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9b21fdf-7mln6\" (UID: \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:12.118797 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.118778    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltp7t\" (UniqueName: \"kubernetes.io/projected/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-kube-api-access-ltp7t\") pod \"maas-api-revoke-sub-c9b21fdf-7mln6\" (UID: \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:12.119261 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.119241    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9b21fdf-7mln6\" (UID: \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:12.121196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.121175    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9b21fdf-7mln6\" (UID: \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:12.143891 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.143863    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltp7t\" (UniqueName: \"kubernetes.io/projected/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-kube-api-access-ltp7t\") pod \"maas-api-revoke-sub-c9b21fdf-7mln6\" (UID: \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:12.145522 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.145500    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:12.250145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.250110    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:12.323645 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.323577    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/959aa0ec-aca6-4538-85b9-f7ee508469fe-kube-api-access-wdn4w\" (UniqueName: \"kubernetes.io/projected/959aa0ec-aca6-4538-85b9-f7ee508469fe-kube-api-access-wdn4w\") pod \"959aa0ec-aca6-4538-85b9-f7ee508469fe\" (UID: \"959aa0ec-aca6-4538-85b9-f7ee508469fe\") "
Nov 02 12:53:12.323868 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.323840    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-openshift-ca\") pod \"959aa0ec-aca6-4538-85b9-f7ee508469fe\" (UID: \"959aa0ec-aca6-4538-85b9-f7ee508469fe\") "
Nov 02 12:53:12.323970 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.323937    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-xks-ca\") pod \"959aa0ec-aca6-4538-85b9-f7ee508469fe\" (UID: \"959aa0ec-aca6-4538-85b9-f7ee508469fe\") "
Nov 02 12:53:12.324618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.324470    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-openshift-ca" pod "959aa0ec-aca6-4538-85b9-f7ee508469fe" (UID: "959aa0ec-aca6-4538-85b9-f7ee508469fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:53:12.326940 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.326910    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-xks-ca" pod "959aa0ec-aca6-4538-85b9-f7ee508469fe" (UID: "959aa0ec-aca6-4538-85b9-f7ee508469fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:53:12.327059 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.326966    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959aa0ec-aca6-4538-85b9-f7ee508469fe-kube-api-access-wdn4w" pod "959aa0ec-aca6-4538-85b9-f7ee508469fe" (UID: "959aa0ec-aca6-4538-85b9-f7ee508469fe"). InnerVolumeSpecName "kube-api-access-wdn4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:12.426144 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.426110    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:12.426144 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.426137    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdn4w\" (UniqueName: \"kubernetes.io/projected/959aa0ec-aca6-4538-85b9-f7ee508469fe-kube-api-access-wdn4w\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:12.426144 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.426148    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/959aa0ec-aca6-4538-85b9-f7ee508469fe-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:12.604213 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:53:12.604116    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b79524e_94e3_4b98_8835_f7f7b59c7b1c.slice/crio-5c370173d4f07b148a2ed76f4a7b83ea5ae5c25257df460b91eedd8f78433d57 WatchSource:0}: Error finding container 5c370173d4f07b148a2ed76f4a7b83ea5ae5c25257df460b91eedd8f78433d57: Status 404 returned error can't find the container with id 5c370173d4f07b148a2ed76f4a7b83ea5ae5c25257df460b91eedd8f78433d57
Nov 02 12:53:12.605894 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.605863    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"]
Nov 02 12:53:12.607863 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.607838    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:12.966258 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.966221    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6" event={"ID":"1b79524e-94e3-4b98-8835-f7f7b59c7b1c","Type":"ContainerStarted","Data":"5c370173d4f07b148a2ed76f4a7b83ea5ae5c25257df460b91eedd8f78433d57"}
Nov 02 12:53:12.968032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.968004    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"
Nov 02 12:53:12.968155 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.968035    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7" event={"ID":"959aa0ec-aca6-4538-85b9-f7ee508469fe","Type":"ContainerDied","Data":"e8634be43e775191958582772926965c611332c8635dbd653dbf3172f3d92e6e"}
Nov 02 12:53:12.968155 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:12.968085    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8634be43e775191958582772926965c611332c8635dbd653dbf3172f3d92e6e"
Nov 02 12:53:13.092834 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.092789    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:13.688150 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.688119    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"]
Nov 02 12:53:13.688785 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.688766    5879 cpu_manager.go:414] "containerMap: removing container" podUID="959aa0ec-aca6-4538-85b9-f7ee508469fe" containerName="revoke-keys"
Nov 02 12:53:13.688878 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.688787    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="959aa0ec-aca6-4538-85b9-f7ee508469fe" containerName="revoke-keys"
Nov 02 12:53:13.688946 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.688912    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="959aa0ec-aca6-4538-85b9-f7ee508469fe" containerName="revoke-keys"
Nov 02 12:53:13.692479 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.692460    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:13.718641 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.718602    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"]
Nov 02 12:53:13.751895 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.751840    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwbwt\" (UniqueName: \"kubernetes.io/projected/60a77211-86ad-4403-80f5-37863af1252f-kube-api-access-xwbwt\") pod \"maas-api-revoke-sub-89eb17ec-dcb2c\" (UID: \"60a77211-86ad-4403-80f5-37863af1252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:13.752134 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.751954    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a77211-86ad-4403-80f5-37863af1252f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89eb17ec-dcb2c\" (UID: \"60a77211-86ad-4403-80f5-37863af1252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:13.752134 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.752061    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a77211-86ad-4403-80f5-37863af1252f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89eb17ec-dcb2c\" (UID: \"60a77211-86ad-4403-80f5-37863af1252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:13.806027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.801904    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:13.853298 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.853255    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a77211-86ad-4403-80f5-37863af1252f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89eb17ec-dcb2c\" (UID: \"60a77211-86ad-4403-80f5-37863af1252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:13.853492 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.853334    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a77211-86ad-4403-80f5-37863af1252f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89eb17ec-dcb2c\" (UID: \"60a77211-86ad-4403-80f5-37863af1252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:13.853492 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.853472    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwbwt\" (UniqueName: \"kubernetes.io/projected/60a77211-86ad-4403-80f5-37863af1252f-kube-api-access-xwbwt\") pod \"maas-api-revoke-sub-89eb17ec-dcb2c\" (UID: \"60a77211-86ad-4403-80f5-37863af1252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:13.854048 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.854023    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a77211-86ad-4403-80f5-37863af1252f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89eb17ec-dcb2c\" (UID: \"60a77211-86ad-4403-80f5-37863af1252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:13.855709 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.855689    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a77211-86ad-4403-80f5-37863af1252f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89eb17ec-dcb2c\" (UID: \"60a77211-86ad-4403-80f5-37863af1252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:13.882523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.882495    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwbwt\" (UniqueName: \"kubernetes.io/projected/60a77211-86ad-4403-80f5-37863af1252f-kube-api-access-xwbwt\") pod \"maas-api-revoke-sub-89eb17ec-dcb2c\" (UID: \"60a77211-86ad-4403-80f5-37863af1252f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:13.934553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.934505    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:13.977181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.977145    5879 generic.go:356] "Generic (PLEG): container finished" podID="1b79524e-94e3-4b98-8835-f7f7b59c7b1c" containerID="e87bfc22abc04a06069a45d6dc8f419e5be5662ae515df091aeb088a8936d634" exitCode=0
Nov 02 12:53:13.977397 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:13.977233    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6" event={"ID":"1b79524e-94e3-4b98-8835-f7f7b59c7b1c","Type":"ContainerDied","Data":"e87bfc22abc04a06069a45d6dc8f419e5be5662ae515df091aeb088a8936d634"}
Nov 02 12:53:14.110133 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:14.110086    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:14.422123 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:53:14.422043    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a77211_86ad_4403_80f5_37863af1252f.slice/crio-d06cd413a086ff7b05bce26f7f2355da5c0672eccf4a1ace73b473b4b96c4a32 WatchSource:0}: Error finding container d06cd413a086ff7b05bce26f7f2355da5c0672eccf4a1ace73b473b4b96c4a32: Status 404 returned error can't find the container with id d06cd413a086ff7b05bce26f7f2355da5c0672eccf4a1ace73b473b4b96c4a32
Nov 02 12:53:14.424262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:14.424233    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"]
Nov 02 12:53:14.824955 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:53:14.824924    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:53:14.989132 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:14.988986    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c" event={"ID":"60a77211-86ad-4403-80f5-37863af1252f","Type":"ContainerStarted","Data":"d5ced8d981c8d5f68261ff7ac78ef96afb28de02f64b633276b59a933e77c47b"}
Nov 02 12:53:14.989606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:14.989574    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c" event={"ID":"60a77211-86ad-4403-80f5-37863af1252f","Type":"ContainerStarted","Data":"d06cd413a086ff7b05bce26f7f2355da5c0672eccf4a1ace73b473b4b96c4a32"}
Nov 02 12:53:15.051478 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.051420    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c" podStartSLOduration=2.051404188 podStartE2EDuration="2.051404188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:15.047748477 +0000 UTC m=+1541.154451357" watchObservedRunningTime="2026-11-02 12:53:15.051404188 +0000 UTC m=+1541.158107039"
Nov 02 12:53:15.166531 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.166505    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:15.275726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.275637    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-kube-api-access-ltp7t\" (UniqueName: \"kubernetes.io/projected/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-kube-api-access-ltp7t\") pod \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\" (UID: \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\") "
Nov 02 12:53:15.275726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.275692    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-xks-ca\") pod \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\" (UID: \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\") "
Nov 02 12:53:15.275974 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.275776    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-openshift-ca\") pod \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\" (UID: \"1b79524e-94e3-4b98-8835-f7f7b59c7b1c\") "
Nov 02 12:53:15.276195 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.276170    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-openshift-ca" pod "1b79524e-94e3-4b98-8835-f7f7b59c7b1c" (UID: "1b79524e-94e3-4b98-8835-f7f7b59c7b1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:53:15.278095 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.278066    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-xks-ca" pod "1b79524e-94e3-4b98-8835-f7f7b59c7b1c" (UID: "1b79524e-94e3-4b98-8835-f7f7b59c7b1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:53:15.278247 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.278229    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-kube-api-access-ltp7t" pod "1b79524e-94e3-4b98-8835-f7f7b59c7b1c" (UID: "1b79524e-94e3-4b98-8835-f7f7b59c7b1c"). InnerVolumeSpecName "kube-api-access-ltp7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:15.377136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.377096    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:15.377136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.377131    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltp7t\" (UniqueName: \"kubernetes.io/projected/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-kube-api-access-ltp7t\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:15.377136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.377144    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b79524e-94e3-4b98-8835-f7f7b59c7b1c-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:15.491018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.486823    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:15.661794 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:15.661744    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:16.229093 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:16.210667    5879 generic.go:356] "Generic (PLEG): container finished" podID="60a77211-86ad-4403-80f5-37863af1252f" containerID="d5ced8d981c8d5f68261ff7ac78ef96afb28de02f64b633276b59a933e77c47b" exitCode=0
Nov 02 12:53:16.229093 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:16.219922    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"
Nov 02 12:53:16.333381 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:16.332718    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c" event={"ID":"60a77211-86ad-4403-80f5-37863af1252f","Type":"ContainerDied","Data":"d5ced8d981c8d5f68261ff7ac78ef96afb28de02f64b633276b59a933e77c47b"}
Nov 02 12:53:16.333381 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:16.332804    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6" event={"ID":"1b79524e-94e3-4b98-8835-f7f7b59c7b1c","Type":"ContainerDied","Data":"5c370173d4f07b148a2ed76f4a7b83ea5ae5c25257df460b91eedd8f78433d57"}
Nov 02 12:53:16.333381 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:16.332831    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c370173d4f07b148a2ed76f4a7b83ea5ae5c25257df460b91eedd8f78433d57"
Nov 02 12:53:16.926021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:16.925312    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.203077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:17.202084    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 12:53:17.203077 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:17.202545    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.395021 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:17.382973    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.449135 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:17.449088    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.463198 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:17.463090    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:17.845842 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:17.845661    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:18.035022 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:17.986927    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 12:53:18.365700 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.345438    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:18.365700 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.345851    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 12:53:18.398426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.398382    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:18.512085 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.510124    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 12:53:18.546583 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.546534    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:18.648449 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.648402    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:18.652247 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.650079    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c" event={"ID":"60a77211-86ad-4403-80f5-37863af1252f","Type":"ContainerDied","Data":"d06cd413a086ff7b05bce26f7f2355da5c0672eccf4a1ace73b473b4b96c4a32"}
Nov 02 12:53:18.663585 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.652654    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:18.687204 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.663833    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d06cd413a086ff7b05bce26f7f2355da5c0672eccf4a1ace73b473b4b96c4a32"
Nov 02 12:53:18.690138 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.690114    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:53:18.690493 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.690469    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:53:18.690736 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.690723    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:53:18.690971 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.690959    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:53:18.833119 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.829134    5879 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:18.833119 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.831591    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 12:53:18.930221 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.919749    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60a77211-86ad-4403-80f5-37863af1252f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a77211-86ad-4403-80f5-37863af1252f-maas-api-xks-ca\") pod \"60a77211-86ad-4403-80f5-37863af1252f\" (UID: \"60a77211-86ad-4403-80f5-37863af1252f\") "
Nov 02 12:53:18.930649 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.930612    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60a77211-86ad-4403-80f5-37863af1252f-kube-api-access-xwbwt\" (UniqueName: \"kubernetes.io/projected/60a77211-86ad-4403-80f5-37863af1252f-kube-api-access-xwbwt\") pod \"60a77211-86ad-4403-80f5-37863af1252f\" (UID: \"60a77211-86ad-4403-80f5-37863af1252f\") "
Nov 02 12:53:18.972020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.947550    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60a77211-86ad-4403-80f5-37863af1252f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a77211-86ad-4403-80f5-37863af1252f-maas-api-openshift-ca\") pod \"60a77211-86ad-4403-80f5-37863af1252f\" (UID: \"60a77211-86ad-4403-80f5-37863af1252f\") "
Nov 02 12:53:18.972726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.972687    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a77211-86ad-4403-80f5-37863af1252f-maas-api-openshift-ca" pod "60a77211-86ad-4403-80f5-37863af1252f" (UID: "60a77211-86ad-4403-80f5-37863af1252f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:53:19.082961 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:18.947348    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:19.200281 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:19.145910    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a77211-86ad-4403-80f5-37863af1252f-kube-api-access-xwbwt" pod "60a77211-86ad-4403-80f5-37863af1252f" (UID: "60a77211-86ad-4403-80f5-37863af1252f"). InnerVolumeSpecName "kube-api-access-xwbwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:19.200281 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:19.146069    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60a77211-86ad-4403-80f5-37863af1252f-maas-api-xks-ca" pod "60a77211-86ad-4403-80f5-37863af1252f" (UID: "60a77211-86ad-4403-80f5-37863af1252f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:53:19.200281 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:19.175328    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 12:53:19.213017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:19.204986    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60a77211-86ad-4403-80f5-37863af1252f-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:19.228135 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:19.228102    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwbwt\" (UniqueName: \"kubernetes.io/projected/60a77211-86ad-4403-80f5-37863af1252f-kube-api-access-xwbwt\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:19.228423 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:19.228403    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60a77211-86ad-4403-80f5-37863af1252f-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:19.454072 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:19.370983    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 12:53:19.575023 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:19.574659    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 12:53:19.732942 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:19.732861    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 12:53:19.751931 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:19.751906    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"
Nov 02 12:53:19.778312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:19.778284    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 12:53:20.287286 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:20.287257    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:20.858377 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:20.858339    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:21.010148 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:21.010108    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:22.148717 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.148683    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"]
Nov 02 12:53:22.149283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.149268    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1b79524e-94e3-4b98-8835-f7f7b59c7b1c" containerName="revoke-keys"
Nov 02 12:53:22.149329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.149283    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b79524e-94e3-4b98-8835-f7f7b59c7b1c" containerName="revoke-keys"
Nov 02 12:53:22.149329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.149300    5879 cpu_manager.go:414] "containerMap: removing container" podUID="60a77211-86ad-4403-80f5-37863af1252f" containerName="revoke-keys"
Nov 02 12:53:22.149329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.149305    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60a77211-86ad-4403-80f5-37863af1252f" containerName="revoke-keys"
Nov 02 12:53:22.149482 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.149416    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="60a77211-86ad-4403-80f5-37863af1252f" containerName="revoke-keys"
Nov 02 12:53:22.149482 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.149434    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b79524e-94e3-4b98-8835-f7f7b59c7b1c" containerName="revoke-keys"
Nov 02 12:53:22.153361 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.153339    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.193069 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.192885    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:22.193693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.193665    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"]
Nov 02 12:53:22.204797 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.204673    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"]
Nov 02 12:53:22.205005 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.204832    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:22.209535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.209488    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"]
Nov 02 12:53:22.226618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.226587    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:53:22.228800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.228774    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:53:22.293419 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.293379    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e73b9582-8zvcj\" (UID: \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:22.293598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.293438    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.293598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.293515    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.293598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.293553    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e73b9582-8zvcj\" (UID: \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:22.293768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.293713    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-569b2\" (UniqueName: \"kubernetes.io/projected/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-kube-api-access-569b2\") pod \"maas-api-revoke-sub-e73b9582-8zvcj\" (UID: \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:22.293811 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.293768    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj6gj\" (UniqueName: \"kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-kube-api-access-dj6gj\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.293847 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.293835    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.293878 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.293867    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.293956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.293937    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.294019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.293967    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.294066 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.294030    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.294119 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.294099    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.395532 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.395502    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.395532 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.395538    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.395806 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.395583    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.395806 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.395648    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.395806 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.395689    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e73b9582-8zvcj\" (UID: \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:22.395806 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.395730    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.395806 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.395753    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.395806 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.395787    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e73b9582-8zvcj\" (UID: \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:22.396147 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.395904    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-569b2\" (UniqueName: \"kubernetes.io/projected/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-kube-api-access-569b2\") pod \"maas-api-revoke-sub-e73b9582-8zvcj\" (UID: \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:22.396147 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.395965    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj6gj\" (UniqueName: \"kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-kube-api-access-dj6gj\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.396147 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.396078    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.396147 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.396130    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.396368 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.395963    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-certs\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.396368 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.396255    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-credential-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.396490 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.396405    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-data\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.396584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.396566    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-socket\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.396842 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.396821    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e73b9582-8zvcj\" (UID: \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:22.397305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.397284    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istiod-ca-cert\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.398192 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.398166    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-envoy\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.398389 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.398374    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-podinfo\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.398446 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.398374    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e73b9582-8zvcj\" (UID: \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:22.420887 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.420820    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-token\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.421143 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.421121    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-569b2\" (UniqueName: \"kubernetes.io/projected/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-kube-api-access-569b2\") pod \"maas-api-revoke-sub-e73b9582-8zvcj\" (UID: \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:22.421189 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.421174    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj6gj\" (UniqueName: \"kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-kube-api-access-dj6gj\") pod \"e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") " pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.500562 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.500527    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:22.536851 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.536817    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:22.838046 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:53:22.838009    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17db8c7_32cf_4a99_9d3c_e87c6b3b69b5.slice/crio-44d5fb105098b64fbe2452a428d5da5ebcee2a8e0d4407b772e199e32fea6300 WatchSource:0}: Error finding container 44d5fb105098b64fbe2452a428d5da5ebcee2a8e0d4407b772e199e32fea6300: Status 404 returned error can't find the container with id 44d5fb105098b64fbe2452a428d5da5ebcee2a8e0d4407b772e199e32fea6300
Nov 02 12:53:22.839626 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.839600    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"]
Nov 02 12:53:22.840742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.840707    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:53:22.840864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.840785    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:53:22.840864 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.840828    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:53:22.879686 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:22.879657    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"]
Nov 02 12:53:23.048020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:23.047949    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:23.301281 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:23.301238    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:23.493577 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:23.493529    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:23.725037 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:23.720470    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:23.864665 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:23.864632    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j" event={"ID":"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5","Type":"ContainerStarted","Data":"b95b5d928133aa7acbb925a30fbf95de474abdf2a9da87282a0566f5ba731d07"}
Nov 02 12:53:23.864920 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:23.864898    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j" event={"ID":"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5","Type":"ContainerStarted","Data":"44d5fb105098b64fbe2452a428d5da5ebcee2a8e0d4407b772e199e32fea6300"}
Nov 02 12:53:23.868835 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:23.868804    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj" event={"ID":"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba","Type":"ContainerStarted","Data":"e5fa1a17e0168c29a943ffc0b1e5b083c9253af160ad6e3768131dbd55c9d45b"}
Nov 02 12:53:23.869245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:23.869224    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj" event={"ID":"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba","Type":"ContainerStarted","Data":"90f505c769da287d2a4106153a7a066c904368f7206b8d34c1dd6c852351ed92"}
Nov 02 12:53:24.103019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:24.102197    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:24.161225 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:24.161107    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j" podStartSLOduration=2.161085569 podStartE2EDuration="2.161085569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:24.159510046 +0000 UTC m=+1550.266212935" watchObservedRunningTime="2026-11-02 12:53:24.161085569 +0000 UTC m=+1550.267788423"
Nov 02 12:53:24.254501 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:24.254436    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj" podStartSLOduration=2.254415703 podStartE2EDuration="2.254415703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:24.246659512 +0000 UTC m=+1550.353362364" watchObservedRunningTime="2026-11-02 12:53:24.254415703 +0000 UTC m=+1550.361118555"
Nov 02 12:53:24.276391 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:24.276355    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:24.503070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:24.502446    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:24.524335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:24.518141    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 02 12:53:24.524335 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:24.518206    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j" podUID="e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 02 12:53:24.559020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:24.558231    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:24.880318 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:24.880286    5879 generic.go:356] "Generic (PLEG): container finished" podID="809fd484-fc6d-4e39-9fe0-2f7d071bd6ba" containerID="e5fa1a17e0168c29a943ffc0b1e5b083c9253af160ad6e3768131dbd55c9d45b" exitCode=0
Nov 02 12:53:24.880953 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:24.880928    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj" event={"ID":"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba","Type":"ContainerDied","Data":"e5fa1a17e0168c29a943ffc0b1e5b083c9253af160ad6e3768131dbd55c9d45b"}
Nov 02 12:53:25.501406 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:25.501368    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 02 12:53:25.501644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:25.501441    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j" podUID="e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 02 12:53:25.668310 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:25.668276    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:26.144968 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.144934    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:26.233072 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.232414    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:26.368766 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.367607    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-xks-ca\") pod \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\" (UID: \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\") "
Nov 02 12:53:26.368766 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.367693    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-openshift-ca\") pod \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\" (UID: \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\") "
Nov 02 12:53:26.368766 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.367810    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-kube-api-access-569b2\" (UniqueName: \"kubernetes.io/projected/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-kube-api-access-569b2\") pod \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\" (UID: \"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba\") "
Nov 02 12:53:26.375667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.370365    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-openshift-ca" pod "809fd484-fc6d-4e39-9fe0-2f7d071bd6ba" (UID: "809fd484-fc6d-4e39-9fe0-2f7d071bd6ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:53:26.375667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.371883    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-kube-api-access-569b2" pod "809fd484-fc6d-4e39-9fe0-2f7d071bd6ba" (UID: "809fd484-fc6d-4e39-9fe0-2f7d071bd6ba"). InnerVolumeSpecName "kube-api-access-569b2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:26.408883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.402704    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-xks-ca" pod "809fd484-fc6d-4e39-9fe0-2f7d071bd6ba" (UID: "809fd484-fc6d-4e39-9fe0-2f7d071bd6ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:53:26.469875 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.469830    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-569b2\" (UniqueName: \"kubernetes.io/projected/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-kube-api-access-569b2\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:26.469875 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.469862    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:26.469875 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.469871    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:26.506068 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.505186    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 02 12:53:26.506068 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.505242    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j" podUID="e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 02 12:53:26.600099 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.600043    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:26.928044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.928012    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"
Nov 02 12:53:26.928044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.928022    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj" event={"ID":"809fd484-fc6d-4e39-9fe0-2f7d071bd6ba","Type":"ContainerDied","Data":"90f505c769da287d2a4106153a7a066c904368f7206b8d34c1dd6c852351ed92"}
Nov 02 12:53:26.928479 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:26.928107    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90f505c769da287d2a4106153a7a066c904368f7206b8d34c1dd6c852351ed92"
Nov 02 12:53:27.046128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:27.046095    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:27.316392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:27.316294    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:27.476184 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:27.476141    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:27.505020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:27.502044    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 02 12:53:27.505020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:27.502120    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j" podUID="e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 02 12:53:28.124705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:28.124670    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:28.313348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:28.313307    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:28.502070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:28.501424    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 02 12:53:28.502070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:28.501487    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j" podUID="e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 02 12:53:29.522618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:29.522316    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 02 12:53:29.522618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:29.522383    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j" podUID="e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 02 12:53:29.531086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:29.524279    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:29.728766 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:29.728728    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:30.138949 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:30.138912    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:30.675584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:30.675554    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:30.676138 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:30.676121    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:30.676839 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:30.676820    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 12:53:30.977683 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:30.977603    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:31.163181 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:31.163146    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:32.235528 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:32.235496    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:32.415620 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:32.415589    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:33.103875 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:33.103840    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:33.829420 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:33.829389    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:33.986869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:33.986839    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:35.511899 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:35.511801    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:35.700489 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:35.700412    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:35.852238 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:35.852153    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:36.319023 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:36.318903    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:37.441668 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:37.441632    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:38.598019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.588364    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"]
Nov 02 12:53:38.599575 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.599540    5879 cpu_manager.go:414] "containerMap: removing container" podUID="809fd484-fc6d-4e39-9fe0-2f7d071bd6ba" containerName="revoke-keys"
Nov 02 12:53:38.599769 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.599752    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809fd484-fc6d-4e39-9fe0-2f7d071bd6ba" containerName="revoke-keys"
Nov 02 12:53:38.600098 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.600079    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="809fd484-fc6d-4e39-9fe0-2f7d071bd6ba" containerName="revoke-keys"
Nov 02 12:53:38.627018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.607280    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:38.627018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.619813    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"]
Nov 02 12:53:38.629234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.629208    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:53:38.644424 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.644282    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:53:38.741443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.741402    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4p8z\" (UniqueName: \"kubernetes.io/projected/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-kube-api-access-r4p8z\") pod \"maas-api-revoke-sub-0eea18e0-92vxx\" (UID: \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:38.741739 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.741717    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eea18e0-92vxx\" (UID: \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:38.742143 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.742120    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eea18e0-92vxx\" (UID: \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:38.843573 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.843536    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eea18e0-92vxx\" (UID: \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:38.843843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.843599    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4p8z\" (UniqueName: \"kubernetes.io/projected/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-kube-api-access-r4p8z\") pod \"maas-api-revoke-sub-0eea18e0-92vxx\" (UID: \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:38.843843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.843656    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eea18e0-92vxx\" (UID: \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:38.844559 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.844529    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eea18e0-92vxx\" (UID: \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:38.847300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.847278    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eea18e0-92vxx\" (UID: \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:38.894329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.894293    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4p8z\" (UniqueName: \"kubernetes.io/projected/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-kube-api-access-r4p8z\") pod \"maas-api-revoke-sub-0eea18e0-92vxx\" (UID: \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:38.975089 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:38.975055    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:39.285075 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:53:39.285042    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e73b6d7_d744_4f73_b1b5_dd3437ad6888.slice/crio-4b688c40a92c7c47f95034777ba20a4dbb2ad2176dec64d83b1ae5a4017f7fba WatchSource:0}: Error finding container 4b688c40a92c7c47f95034777ba20a4dbb2ad2176dec64d83b1ae5a4017f7fba: Status 404 returned error can't find the container with id 4b688c40a92c7c47f95034777ba20a4dbb2ad2176dec64d83b1ae5a4017f7fba
Nov 02 12:53:39.286586 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:39.286562    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"]
Nov 02 12:53:39.471369 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:39.471336    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:40.057312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:40.057261    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx" event={"ID":"6e73b6d7-d744-4f73-b1b5-dd3437ad6888","Type":"ContainerStarted","Data":"fa45a2ff95c0734f8d30ab509f4d7044b90beef66552864d77fa59c9e785991d"}
Nov 02 12:53:40.057795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:40.057338    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx" event={"ID":"6e73b6d7-d744-4f73-b1b5-dd3437ad6888","Type":"ContainerStarted","Data":"4b688c40a92c7c47f95034777ba20a4dbb2ad2176dec64d83b1ae5a4017f7fba"}
Nov 02 12:53:40.117087 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:40.117027    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx" podStartSLOduration=2.117011614 podStartE2EDuration="2.117011614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:40.113876662 +0000 UTC m=+1566.220579514" watchObservedRunningTime="2026-11-02 12:53:40.117011614 +0000 UTC m=+1566.223714461"
Nov 02 12:53:41.066186 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:41.066155    5879 generic.go:356] "Generic (PLEG): container finished" podID="6e73b6d7-d744-4f73-b1b5-dd3437ad6888" containerID="fa45a2ff95c0734f8d30ab509f4d7044b90beef66552864d77fa59c9e785991d" exitCode=0
Nov 02 12:53:41.066599 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:41.066247    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx" event={"ID":"6e73b6d7-d744-4f73-b1b5-dd3437ad6888","Type":"ContainerDied","Data":"fa45a2ff95c0734f8d30ab509f4d7044b90beef66552864d77fa59c9e785991d"}
Nov 02 12:53:41.263432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:41.263377    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:42.239029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:42.239003    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:42.394962 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:42.394926    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-kube-api-access-r4p8z\" (UniqueName: \"kubernetes.io/projected/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-kube-api-access-r4p8z\") pod \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\" (UID: \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\") "
Nov 02 12:53:42.395185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:42.394977    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-openshift-ca\") pod \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\" (UID: \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\") "
Nov 02 12:53:42.395185 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:42.395128    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-xks-ca\") pod \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\" (UID: \"6e73b6d7-d744-4f73-b1b5-dd3437ad6888\") "
Nov 02 12:53:42.395361 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:42.395338    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-openshift-ca" pod "6e73b6d7-d744-4f73-b1b5-dd3437ad6888" (UID: "6e73b6d7-d744-4f73-b1b5-dd3437ad6888"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:53:42.395671 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:42.395655    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:42.397392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:42.397358    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-xks-ca" pod "6e73b6d7-d744-4f73-b1b5-dd3437ad6888" (UID: "6e73b6d7-d744-4f73-b1b5-dd3437ad6888"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:53:42.397392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:42.397376    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-kube-api-access-r4p8z" pod "6e73b6d7-d744-4f73-b1b5-dd3437ad6888" (UID: "6e73b6d7-d744-4f73-b1b5-dd3437ad6888"). InnerVolumeSpecName "kube-api-access-r4p8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:53:42.496852 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:42.496823    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4p8z\" (UniqueName: \"kubernetes.io/projected/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-kube-api-access-r4p8z\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:42.496852 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:42.496849    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e73b6d7-d744-4f73-b1b5-dd3437ad6888-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:53:43.027954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:43.027920    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:43.080725 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:43.080698    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"
Nov 02 12:53:43.080890 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:43.080727    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx" event={"ID":"6e73b6d7-d744-4f73-b1b5-dd3437ad6888","Type":"ContainerDied","Data":"4b688c40a92c7c47f95034777ba20a4dbb2ad2176dec64d83b1ae5a4017f7fba"}
Nov 02 12:53:43.080890 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:43.080775    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b688c40a92c7c47f95034777ba20a4dbb2ad2176dec64d83b1ae5a4017f7fba"
Nov 02 12:53:43.175309 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:43.175271    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:44.749074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.749036    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"]
Nov 02 12:53:44.750078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.750045    5879 cpu_manager.go:414] "containerMap: removing container" podUID="6e73b6d7-d744-4f73-b1b5-dd3437ad6888" containerName="revoke-keys"
Nov 02 12:53:44.750078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.750071    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e73b6d7-d744-4f73-b1b5-dd3437ad6888" containerName="revoke-keys"
Nov 02 12:53:44.750305 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.750288    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e73b6d7-d744-4f73-b1b5-dd3437ad6888" containerName="revoke-keys"
Nov 02 12:53:44.754832 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.754809    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:44.789321 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.789284    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"]
Nov 02 12:53:44.837721 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:53:44.837687    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:53:44.945108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.945071    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:44.945328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.945222    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86zhp\" (UniqueName: \"kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-kube-api-access-86zhp\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:44.945328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.945254    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:44.945448 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.945380    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:44.945448 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.945430    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:44.945448 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.945447    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:44.945606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.945548    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:44.945606 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.945598    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:44.945712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:44.945643    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.047393 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.047305    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.047393 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.047359    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.047393 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.047380    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.047696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.047449    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.047696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.047481    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.047696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.047530    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.047696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.047565    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.047696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.047690    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86zhp\" (UniqueName: \"kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-kube-api-access-86zhp\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.047940 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.047735    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.047940 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.047748    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-credential-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.047940 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.047859    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-data\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.048087 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.047948    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-socket\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.048146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.048133    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-certs\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.048359 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.048338    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istiod-ca-cert\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.050214 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.050194    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-envoy\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.050302 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.050262    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-podinfo\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.084520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.084484    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-token\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.084727 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.084539    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86zhp\" (UniqueName: \"kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-kube-api-access-86zhp\") pod \"e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") " pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.373788 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.373753    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:45.405294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.405255    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:45.559024 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.554417    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:45.710876 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:53:45.710839    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d422059_e2fc_48d1_bc39_d0ed04ab8b0b.slice/crio-59f556f2c8b893a0706604e518a3cdb96ad76d045d81800b909db6317c26324c WatchSource:0}: Error finding container 59f556f2c8b893a0706604e518a3cdb96ad76d045d81800b909db6317c26324c: Status 404 returned error can't find the container with id 59f556f2c8b893a0706604e518a3cdb96ad76d045d81800b909db6317c26324c
Nov 02 12:53:45.711480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.711460    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"]
Nov 02 12:53:45.713769 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.713735    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:53:45.713885 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.713819    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:53:45.713885 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:45.713867    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 12:53:46.106594 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:46.106490    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" event={"ID":"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b","Type":"ContainerStarted","Data":"b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977"}
Nov 02 12:53:46.106594 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:46.106564    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" event={"ID":"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b","Type":"ContainerStarted","Data":"59f556f2c8b893a0706604e518a3cdb96ad76d045d81800b909db6317c26324c"}
Nov 02 12:53:46.187281 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:46.187196    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" podStartSLOduration=2.187175158 podStartE2EDuration="2.187175158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:53:46.17279392 +0000 UTC m=+1572.279496772" watchObservedRunningTime="2026-11-02 12:53:46.187175158 +0000 UTC m=+1572.293878012"
Nov 02 12:53:46.374083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:46.374041    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:46.375685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:46.375656    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:46.375803 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:46.375710    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" podUID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 12:53:47.374223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:47.374193    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:47.374619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:47.374242    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" podUID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 12:53:47.583956 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:47.578131    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:48.375210 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:48.375087    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:48.375644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:48.375250    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" podUID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 12:53:49.374410 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:49.374374    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:49.374594 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:49.374447    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" podUID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 12:53:49.514115 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:49.514079    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:49.707454 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:49.707364    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:49.866416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:49.866383    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:50.281298 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:50.281251    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:50.374551 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:50.374515    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:50.374750 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:50.374590    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" podUID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 12:53:50.643688 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:50.643609    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:51.374714 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:51.374675    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:51.374916 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:51.374754    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" podUID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 12:53:51.860147 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:51.857804    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:52.377903 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:52.377332    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Nov 02 12:53:52.377903 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:52.377415    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" podUID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Nov 02 12:53:53.379111 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:53.379077    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:53.379636 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:53.379595    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:53.380713 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:53.380684    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 12:53:53.980571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:53.980448    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:54.871448 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:54.871407    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:55.032201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:55.032165    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:56.267221 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:56.267183    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:56.853088 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:56.853052    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:57.000220 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:57.000188    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:53:58.445035 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:53:58.444961    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:01.019145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:01.019106    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:02.766376 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:02.766311    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:03.155769 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:03.155731    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:04.390242 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:04.390209    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:04.994237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:04.994206    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:05.189287 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:05.189246    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:05.604343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:05.604306    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:06.972514 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:06.972430    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:07.565245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:07.565193    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:08.012809 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:08.012778    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:08.644444 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:08.644408    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:08.883469 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:08.883420    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:09.055201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:09.055164    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:09.279604 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:09.279500    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:09.421336 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:09.421298    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:11.131542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.131494    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:11.271777 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.271747    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:11.382290 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.381965    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"]
Nov 02 12:54:11.537368 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.537333    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"]
Nov 02 12:54:11.537676 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.537639    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"
Nov 02 12:54:11.561725 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.561692    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-74ba71\""
Nov 02 12:54:11.569063 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.569033    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-74ba71-dockercfg-dcld2\""
Nov 02 12:54:11.679569 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.679475    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbp86\" (UniqueName: \"kubernetes.io/projected/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-kube-api-access-wbp86\") pod \"payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj\" (UID: \"99f9f86a-b60e-4b2b-8535-c28fd981e6e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"
Nov 02 12:54:11.679757 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.679679    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj\" (UID: \"99f9f86a-b60e-4b2b-8535-c28fd981e6e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"
Nov 02 12:54:11.780770 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.780728    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbp86\" (UniqueName: \"kubernetes.io/projected/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-kube-api-access-wbp86\") pod \"payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj\" (UID: \"99f9f86a-b60e-4b2b-8535-c28fd981e6e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"
Nov 02 12:54:11.780959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.780906    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj\" (UID: \"99f9f86a-b60e-4b2b-8535-c28fd981e6e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"
Nov 02 12:54:11.781651 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.781632    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj\" (UID: \"99f9f86a-b60e-4b2b-8535-c28fd981e6e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"
Nov 02 12:54:11.806333 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.806301    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbp86\" (UniqueName: \"kubernetes.io/projected/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-kube-api-access-wbp86\") pod \"payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj\" (UID: \"99f9f86a-b60e-4b2b-8535-c28fd981e6e6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"
Nov 02 12:54:11.861414 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:11.861383    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"
Nov 02 12:54:12.171407 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:54:12.171376    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f9f86a_b60e_4b2b_8535_c28fd981e6e6.slice/crio-ce03e33abd4f7cec1abb2688a62cd3fce0fd551d99173c3b2c554979b8bd2aff WatchSource:0}: Error finding container ce03e33abd4f7cec1abb2688a62cd3fce0fd551d99173c3b2c554979b8bd2aff: Status 404 returned error can't find the container with id ce03e33abd4f7cec1abb2688a62cd3fce0fd551d99173c3b2c554979b8bd2aff
Nov 02 12:54:12.172553 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:12.172527    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"]
Nov 02 12:54:12.533849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:12.533753    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj" event={"ID":"99f9f86a-b60e-4b2b-8535-c28fd981e6e6","Type":"ContainerStarted","Data":"873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57"}
Nov 02 12:54:12.533849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:12.533815    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj" event={"ID":"99f9f86a-b60e-4b2b-8535-c28fd981e6e6","Type":"ContainerStarted","Data":"ce03e33abd4f7cec1abb2688a62cd3fce0fd551d99173c3b2c554979b8bd2aff"}
Nov 02 12:54:12.533849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:12.533844    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"
Nov 02 12:54:12.589081 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:12.589020    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj" podStartSLOduration=1.588984014 podStartE2EDuration="1.588984014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:12.585540243 +0000 UTC m=+1598.692243096" watchObservedRunningTime="2026-11-02 12:54:12.588984014 +0000 UTC m=+1598.695686865"
Nov 02 12:54:12.642514 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:12.642479    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:12.828075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:12.827346    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:13.420559 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:13.420523    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:13.868063 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:13.868029    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:14.304361 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:14.304258    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:14.822817 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:54:14.822779    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:54:14.823354 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:14.823004    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:14.977709 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:14.977667    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:15.277613 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:15.277579    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:15.435701 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:15.435668    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:15.589286 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:15.589193    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:15.866686 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:15.866661    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:16.280043 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:16.279899    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:16.840288 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:16.840242    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:17.039532 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.039485    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:17.335060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.334541    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:17.462780 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.462742    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"]
Nov 02 12:54:17.467357 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.467338    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"
Nov 02 12:54:17.500355 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.500322    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-1aa18c-dockercfg-cp7bb\""
Nov 02 12:54:17.504973 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.504933    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3923b227-4f36-4de9-9ef7-b83ea7b28dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx\" (UID: \"3923b227-4f36-4de9-9ef7-b83ea7b28dae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"
Nov 02 12:54:17.505498 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.505469    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh6sp\" (UniqueName: \"kubernetes.io/projected/3923b227-4f36-4de9-9ef7-b83ea7b28dae-kube-api-access-kh6sp\") pod \"payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx\" (UID: \"3923b227-4f36-4de9-9ef7-b83ea7b28dae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"
Nov 02 12:54:17.506598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.506563    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"]
Nov 02 12:54:17.519012 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.518962    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-1aa18c\""
Nov 02 12:54:17.652030 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.647568    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:17.686050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.676711    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh6sp\" (UniqueName: \"kubernetes.io/projected/3923b227-4f36-4de9-9ef7-b83ea7b28dae-kube-api-access-kh6sp\") pod \"payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx\" (UID: \"3923b227-4f36-4de9-9ef7-b83ea7b28dae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"
Nov 02 12:54:17.686050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.676886    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3923b227-4f36-4de9-9ef7-b83ea7b28dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx\" (UID: \"3923b227-4f36-4de9-9ef7-b83ea7b28dae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"
Nov 02 12:54:17.686050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.678345    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3923b227-4f36-4de9-9ef7-b83ea7b28dae-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx\" (UID: \"3923b227-4f36-4de9-9ef7-b83ea7b28dae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"
Nov 02 12:54:17.686050 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.685622    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"]
Nov 02 12:54:17.697457 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.697381    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"
Nov 02 12:54:17.710884 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.710857    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"]
Nov 02 12:54:17.717463 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.717437    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh6sp\" (UniqueName: \"kubernetes.io/projected/3923b227-4f36-4de9-9ef7-b83ea7b28dae-kube-api-access-kh6sp\") pod \"payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx\" (UID: \"3923b227-4f36-4de9-9ef7-b83ea7b28dae\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"
Nov 02 12:54:17.782525 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.782488    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df2rl\" (UniqueName: \"kubernetes.io/projected/98bb8b0a-8ab1-4150-af7d-42d8429199d7-kube-api-access-df2rl\") pod \"payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv\" (UID: \"98bb8b0a-8ab1-4150-af7d-42d8429199d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"
Nov 02 12:54:17.796955 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.791977    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bb8b0a-8ab1-4150-af7d-42d8429199d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv\" (UID: \"98bb8b0a-8ab1-4150-af7d-42d8429199d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"
Nov 02 12:54:17.797556 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.797525    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"
Nov 02 12:54:17.894128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.894073    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df2rl\" (UniqueName: \"kubernetes.io/projected/98bb8b0a-8ab1-4150-af7d-42d8429199d7-kube-api-access-df2rl\") pod \"payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv\" (UID: \"98bb8b0a-8ab1-4150-af7d-42d8429199d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"
Nov 02 12:54:17.894128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.894135    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bb8b0a-8ab1-4150-af7d-42d8429199d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv\" (UID: \"98bb8b0a-8ab1-4150-af7d-42d8429199d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"
Nov 02 12:54:17.895179 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.895147    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bb8b0a-8ab1-4150-af7d-42d8429199d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv\" (UID: \"98bb8b0a-8ab1-4150-af7d-42d8429199d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"
Nov 02 12:54:17.929806 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.929768    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df2rl\" (UniqueName: \"kubernetes.io/projected/98bb8b0a-8ab1-4150-af7d-42d8429199d7-kube-api-access-df2rl\") pod \"payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv\" (UID: \"98bb8b0a-8ab1-4150-af7d-42d8429199d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"
Nov 02 12:54:17.984020 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:17.982352    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:18.057162 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.057125    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"
Nov 02 12:54:18.175760 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.175615    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"]
Nov 02 12:54:18.430847 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:54:18.430811    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bb8b0a_8ab1_4150_af7d_42d8429199d7.slice/crio-02cd7e23debad1663a4b39868640675eaccc6581914582d50f03f43a7843622b WatchSource:0}: Error finding container 02cd7e23debad1663a4b39868640675eaccc6581914582d50f03f43a7843622b: Status 404 returned error can't find the container with id 02cd7e23debad1663a4b39868640675eaccc6581914582d50f03f43a7843622b
Nov 02 12:54:18.432065 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.432039    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"]
Nov 02 12:54:18.594983 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.594952    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:18.680523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.680431    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx" event={"ID":"3923b227-4f36-4de9-9ef7-b83ea7b28dae","Type":"ContainerStarted","Data":"c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85"}
Nov 02 12:54:18.680523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.680500    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx" event={"ID":"3923b227-4f36-4de9-9ef7-b83ea7b28dae","Type":"ContainerStarted","Data":"bf9da386f77f55852996a1b9b59aa8b9a0bfb70ce16bb4bbc46f88c3ba6a4174"}
Nov 02 12:54:18.680751 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.680581    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"
Nov 02 12:54:18.682699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.682658    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv" event={"ID":"98bb8b0a-8ab1-4150-af7d-42d8429199d7","Type":"ContainerStarted","Data":"0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb"}
Nov 02 12:54:18.682845 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.682702    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv" event={"ID":"98bb8b0a-8ab1-4150-af7d-42d8429199d7","Type":"ContainerStarted","Data":"02cd7e23debad1663a4b39868640675eaccc6581914582d50f03f43a7843622b"}
Nov 02 12:54:18.682845 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.682725    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"
Nov 02 12:54:18.735954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.735899    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx" podStartSLOduration=1.735883978 podStartE2EDuration="1.735883978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:18.732658211 +0000 UTC m=+1604.839361063" watchObservedRunningTime="2026-11-02 12:54:18.735883978 +0000 UTC m=+1604.842586829"
Nov 02 12:54:18.757739 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.757710    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:18.802201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:18.802151    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv" podStartSLOduration=1.802137377 podStartE2EDuration="1.802137377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:18.799215064 +0000 UTC m=+1604.905917918" watchObservedRunningTime="2026-11-02 12:54:18.802137377 +0000 UTC m=+1604.908840228"
Nov 02 12:54:20.497769 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:20.497739    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:21.023037 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.023005    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:21.182167 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.182117    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:21.363883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.363850    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:21.448220 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.448189    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"]
Nov 02 12:54:21.452725 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.452704    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"
Nov 02 12:54:21.457247 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.457220    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"]
Nov 02 12:54:21.461910 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.461891    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"
Nov 02 12:54:21.471255 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.471229    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-1e14d4\""
Nov 02 12:54:21.471459 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.471436    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-1e14d4-dockercfg-9z2wq\""
Nov 02 12:54:21.491618 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.491591    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"]
Nov 02 12:54:21.506820 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.506791    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"]
Nov 02 12:54:21.540452 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.540420    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:21.559947 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.559909    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c33d267-6b2a-41e0-ba1c-685284c7137e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz\" (UID: \"0c33d267-6b2a-41e0-ba1c-685284c7137e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"
Nov 02 12:54:21.560201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.559977    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2zr\" (UniqueName: \"kubernetes.io/projected/0c33d267-6b2a-41e0-ba1c-685284c7137e-kube-api-access-lr2zr\") pod \"payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz\" (UID: \"0c33d267-6b2a-41e0-ba1c-685284c7137e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"
Nov 02 12:54:21.560201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.560104    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8\" (UID: \"63f632d1-51bb-4f53-b7eb-afc0dff0cce2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"
Nov 02 12:54:21.560329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.560238    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgw5v\" (UniqueName: \"kubernetes.io/projected/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-kube-api-access-dgw5v\") pod \"payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8\" (UID: \"63f632d1-51bb-4f53-b7eb-afc0dff0cce2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"
Nov 02 12:54:21.661988 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.661887    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c33d267-6b2a-41e0-ba1c-685284c7137e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz\" (UID: \"0c33d267-6b2a-41e0-ba1c-685284c7137e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"
Nov 02 12:54:21.661988 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.661950    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2zr\" (UniqueName: \"kubernetes.io/projected/0c33d267-6b2a-41e0-ba1c-685284c7137e-kube-api-access-lr2zr\") pod \"payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz\" (UID: \"0c33d267-6b2a-41e0-ba1c-685284c7137e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"
Nov 02 12:54:21.662269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.662059    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8\" (UID: \"63f632d1-51bb-4f53-b7eb-afc0dff0cce2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"
Nov 02 12:54:21.662269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.662191    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgw5v\" (UniqueName: \"kubernetes.io/projected/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-kube-api-access-dgw5v\") pod \"payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8\" (UID: \"63f632d1-51bb-4f53-b7eb-afc0dff0cce2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"
Nov 02 12:54:21.662756 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.662737    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c33d267-6b2a-41e0-ba1c-685284c7137e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz\" (UID: \"0c33d267-6b2a-41e0-ba1c-685284c7137e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"
Nov 02 12:54:21.663280 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.663257    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8\" (UID: \"63f632d1-51bb-4f53-b7eb-afc0dff0cce2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"
Nov 02 12:54:21.710673 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.710640    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgw5v\" (UniqueName: \"kubernetes.io/projected/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-kube-api-access-dgw5v\") pod \"payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8\" (UID: \"63f632d1-51bb-4f53-b7eb-afc0dff0cce2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"
Nov 02 12:54:21.730152 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.730122    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2zr\" (UniqueName: \"kubernetes.io/projected/0c33d267-6b2a-41e0-ba1c-685284c7137e-kube-api-access-lr2zr\") pod \"payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz\" (UID: \"0c33d267-6b2a-41e0-ba1c-685284c7137e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"
Nov 02 12:54:21.769371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.769338    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"
Nov 02 12:54:21.786527 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:21.786498    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"
Nov 02 12:54:22.144010 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:54:22.143962    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c33d267_6b2a_41e0_ba1c_685284c7137e.slice/crio-c9ca2c9e58154790fc0c6c9da27dcb8e23849f7c040907863d44126149680ac4 WatchSource:0}: Error finding container c9ca2c9e58154790fc0c6c9da27dcb8e23849f7c040907863d44126149680ac4: Status 404 returned error can't find the container with id c9ca2c9e58154790fc0c6c9da27dcb8e23849f7c040907863d44126149680ac4
Nov 02 12:54:22.151835 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:22.151808    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"]
Nov 02 12:54:22.159686 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:22.159660    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"]
Nov 02 12:54:22.159799 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:54:22.159730    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f632d1_51bb_4f53_b7eb_afc0dff0cce2.slice/crio-ba81f34a06d513da2314691ed78dafd0863caad9aa8332e803b0f739e5b541e0 WatchSource:0}: Error finding container ba81f34a06d513da2314691ed78dafd0863caad9aa8332e803b0f739e5b541e0: Status 404 returned error can't find the container with id ba81f34a06d513da2314691ed78dafd0863caad9aa8332e803b0f739e5b541e0
Nov 02 12:54:22.723229 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:22.723194    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8" event={"ID":"63f632d1-51bb-4f53-b7eb-afc0dff0cce2","Type":"ContainerStarted","Data":"8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd"}
Nov 02 12:54:22.723681 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:22.723257    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8" event={"ID":"63f632d1-51bb-4f53-b7eb-afc0dff0cce2","Type":"ContainerStarted","Data":"ba81f34a06d513da2314691ed78dafd0863caad9aa8332e803b0f739e5b541e0"}
Nov 02 12:54:22.723681 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:22.723288    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"
Nov 02 12:54:22.725309 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:22.725282    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz" event={"ID":"0c33d267-6b2a-41e0-ba1c-685284c7137e","Type":"ContainerStarted","Data":"40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146"}
Nov 02 12:54:22.725456 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:22.725334    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz" event={"ID":"0c33d267-6b2a-41e0-ba1c-685284c7137e","Type":"ContainerStarted","Data":"c9ca2c9e58154790fc0c6c9da27dcb8e23849f7c040907863d44126149680ac4"}
Nov 02 12:54:22.725456 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:22.725362    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"
Nov 02 12:54:22.772341 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:22.772296    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8" podStartSLOduration=1.772282825 podStartE2EDuration="1.772282825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:22.77053576 +0000 UTC m=+1608.877238607" watchObservedRunningTime="2026-11-02 12:54:22.772282825 +0000 UTC m=+1608.878985689"
Nov 02 12:54:23.171157 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:23.171114    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:23.544377 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:23.544297    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"
Nov 02 12:54:23.592149 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:23.592090    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz" podStartSLOduration=2.592062704 podStartE2EDuration="2.592062704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:22.839508431 +0000 UTC m=+1608.946211285" watchObservedRunningTime="2026-11-02 12:54:23.592062704 +0000 UTC m=+1609.698765556"
Nov 02 12:54:23.964031 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:23.963958    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:24.127363 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.127333    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:24.197034 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.196981    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"]
Nov 02 12:54:24.201281 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.201249    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"
Nov 02 12:54:24.228177 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.228094    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"]
Nov 02 12:54:24.314087 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.296311    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-6416f0-dockercfg-mmgb2\""
Nov 02 12:54:24.314087 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.296398    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-6416f0\""
Nov 02 12:54:24.331487 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.327252    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtfqq\" (UniqueName: \"kubernetes.io/projected/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-kube-api-access-wtfqq\") pod \"payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp\" (UID: \"a1e903ab-013b-42e2-9a21-3f94d1f7e79e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"
Nov 02 12:54:24.331487 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.327425    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp\" (UID: \"a1e903ab-013b-42e2-9a21-3f94d1f7e79e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"
Nov 02 12:54:24.430689 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.428895    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtfqq\" (UniqueName: \"kubernetes.io/projected/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-kube-api-access-wtfqq\") pod \"payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp\" (UID: \"a1e903ab-013b-42e2-9a21-3f94d1f7e79e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"
Nov 02 12:54:24.430689 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.429080    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp\" (UID: \"a1e903ab-013b-42e2-9a21-3f94d1f7e79e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"
Nov 02 12:54:24.430689 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.430037    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp\" (UID: \"a1e903ab-013b-42e2-9a21-3f94d1f7e79e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"
Nov 02 12:54:24.463078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.463042    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtfqq\" (UniqueName: \"kubernetes.io/projected/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-kube-api-access-wtfqq\") pod \"payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp\" (UID: \"a1e903ab-013b-42e2-9a21-3f94d1f7e79e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"
Nov 02 12:54:24.516841 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.516717    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"
Nov 02 12:54:24.886129 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:24.884853    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"]
Nov 02 12:54:25.132209 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:25.132126    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:25.750822 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:25.750778    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp" event={"ID":"a1e903ab-013b-42e2-9a21-3f94d1f7e79e","Type":"ContainerStarted","Data":"504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9"}
Nov 02 12:54:25.751055 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:25.750865    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp" event={"ID":"a1e903ab-013b-42e2-9a21-3f94d1f7e79e","Type":"ContainerStarted","Data":"629701a0631b6758656cfe8381eda36a43416ce71b08c312e3e699a307a71231"}
Nov 02 12:54:25.751055 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:25.750889    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"
Nov 02 12:54:25.803510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:25.803453    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp" podStartSLOduration=1.803438007 podStartE2EDuration="1.803438007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:25.798591649 +0000 UTC m=+1611.905294499" watchObservedRunningTime="2026-11-02 12:54:25.803438007 +0000 UTC m=+1611.910140861"
Nov 02 12:54:27.057326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:27.057285    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:28.824542 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:28.824507    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:29.694222 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:29.694192    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"
Nov 02 12:54:29.694387 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:29.694265    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"
Nov 02 12:54:30.510397 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:30.510364    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:32.103297 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:32.103264    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:32.241869 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:32.241833    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:32.423823 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:32.423796    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:33.736053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:33.736013    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"
Nov 02 12:54:33.736440 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:33.736361    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"
Nov 02 12:54:33.878080 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:33.878047    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:35.571402 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:35.571365    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:36.461662 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:36.461630    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:36.761304 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:36.761220    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"
Nov 02 12:54:37.599073 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:37.599041    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:37.750584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:37.750545    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:39.455194 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:39.455159    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"]
Nov 02 12:54:39.455729 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:39.455367    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy" containerID="cri-o://09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc" gracePeriod=30
Nov 02 12:54:39.521027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:39.520963    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:41.322980 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:41.322935    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:43.121432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:43.121392    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:43.255759 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:43.255727    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:43.403405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:43.403327    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:44.743174 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.743114    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:54:44.826308 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:54:44.826272    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:54:44.912552 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.912512    5879 generic.go:356] "Generic (PLEG): container finished" podID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerID="09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc" exitCode=0
Nov 02 12:54:44.912843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.912577    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"
Nov 02 12:54:44.912843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.912589    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" event={"ID":"50b970dd-6fcd-4e7a-913a-03abc88d0969","Type":"ContainerDied","Data":"09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc"}
Nov 02 12:54:44.912843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.912677    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt" event={"ID":"50b970dd-6fcd-4e7a-913a-03abc88d0969","Type":"ContainerDied","Data":"4a58064168c841f7524622ee25d35eaa49b1d3c16e92ddd541e5794e345fe4c1"}
Nov 02 12:54:44.912843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.912710    5879 scope.go:122] "RemoveContainer" containerID="09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc"
Nov 02 12:54:44.918455 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.918429    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-certs\") pod \"50b970dd-6fcd-4e7a-913a-03abc88d0969\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") "
Nov 02 12:54:44.918584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.918492    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-credential-socket\") pod \"50b970dd-6fcd-4e7a-913a-03abc88d0969\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") "
Nov 02 12:54:44.918584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.918537    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-data\") pod \"50b970dd-6fcd-4e7a-913a-03abc88d0969\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") "
Nov 02 12:54:44.918584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.918566    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b970dd-6fcd-4e7a-913a-03abc88d0969-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b970dd-6fcd-4e7a-913a-03abc88d0969-istiod-ca-cert\") pod \"50b970dd-6fcd-4e7a-913a-03abc88d0969\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") "
Nov 02 12:54:44.918757 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.918626    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-socket\") pod \"50b970dd-6fcd-4e7a-913a-03abc88d0969\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") "
Nov 02 12:54:44.918757 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.918724    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-token\" (UniqueName: \"kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-token\") pod \"50b970dd-6fcd-4e7a-913a-03abc88d0969\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") "
Nov 02 12:54:44.918865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.918769    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-envoy\") pod \"50b970dd-6fcd-4e7a-913a-03abc88d0969\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") "
Nov 02 12:54:44.918865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.918806    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-podinfo\") pod \"50b970dd-6fcd-4e7a-913a-03abc88d0969\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") "
Nov 02 12:54:44.918865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.918809    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-certs" pod "50b970dd-6fcd-4e7a-913a-03abc88d0969" (UID: "50b970dd-6fcd-4e7a-913a-03abc88d0969"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:54:44.918865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.918844    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-credential-socket" pod "50b970dd-6fcd-4e7a-913a-03abc88d0969" (UID: "50b970dd-6fcd-4e7a-913a-03abc88d0969"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:54:44.918865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.918857    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-data" pod "50b970dd-6fcd-4e7a-913a-03abc88d0969" (UID: "50b970dd-6fcd-4e7a-913a-03abc88d0969"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:54:44.919135 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.918899    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-kube-api-access-h4mwc\" (UniqueName: \"kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-kube-api-access-h4mwc\") pod \"50b970dd-6fcd-4e7a-913a-03abc88d0969\" (UID: \"50b970dd-6fcd-4e7a-913a-03abc88d0969\") "
Nov 02 12:54:44.919135 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.919051    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-socket" pod "50b970dd-6fcd-4e7a-913a-03abc88d0969" (UID: "50b970dd-6fcd-4e7a-913a-03abc88d0969"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:54:44.919234 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.919181    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b970dd-6fcd-4e7a-913a-03abc88d0969-istiod-ca-cert" pod "50b970dd-6fcd-4e7a-913a-03abc88d0969" (UID: "50b970dd-6fcd-4e7a-913a-03abc88d0969"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:54:44.919570 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.919551    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:54:44.919696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.919682    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:54:44.920036 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.919962    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:54:44.920036 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.919983    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50b970dd-6fcd-4e7a-913a-03abc88d0969-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:54:44.920036 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.920013    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:54:44.923158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.923104    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-envoy" pod "50b970dd-6fcd-4e7a-913a-03abc88d0969" (UID: "50b970dd-6fcd-4e7a-913a-03abc88d0969"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:54:44.923158 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.923150    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-token" pod "50b970dd-6fcd-4e7a-913a-03abc88d0969" (UID: "50b970dd-6fcd-4e7a-913a-03abc88d0969"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:54:44.923339 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.923182    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-kube-api-access-h4mwc" pod "50b970dd-6fcd-4e7a-913a-03abc88d0969" (UID: "50b970dd-6fcd-4e7a-913a-03abc88d0969"). InnerVolumeSpecName "kube-api-access-h4mwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:54:44.923339 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.923241    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-podinfo" pod "50b970dd-6fcd-4e7a-913a-03abc88d0969" (UID: "50b970dd-6fcd-4e7a-913a-03abc88d0969"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:54:44.925757 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.925740    5879 scope.go:122] "RemoveContainer" containerID="09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc"
Nov 02 12:54:44.926067 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:54:44.926045    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc\": container with ID starting with 09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc not found: ID does not exist" containerID="09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc"
Nov 02 12:54:44.926156 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:44.926073    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc"} err="failed to get container status \"09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc\": rpc error: code = NotFound desc = could not find container \"09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc\": container with ID starting with 09fc337fd0e02bc321740e35e44f18ab891232f7705227c1c3a8b69aa60d83bc not found: ID does not exist"
Nov 02 12:54:45.021852 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:45.021763    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:54:45.021852 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:45.021794    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:54:45.021852 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:45.021806    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50b970dd-6fcd-4e7a-913a-03abc88d0969-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:54:45.021852 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:45.021818    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4mwc\" (UniqueName: \"kubernetes.io/projected/50b970dd-6fcd-4e7a-913a-03abc88d0969-kube-api-access-h4mwc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:54:45.298524 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:45.298412    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"]
Nov 02 12:54:45.340019 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:45.339040    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-fx2tt"]
Nov 02 12:54:45.431283 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:45.431255    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:45.757745 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:45.757702    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:46.018352 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:46.018274    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" path="/var/lib/kubelet/pods/50b970dd-6fcd-4e7a-913a-03abc88d0969/volumes"
Nov 02 12:54:46.223532 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:46.223497    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:46.741358 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:46.741327    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:47.326235 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:47.326160    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:47.549096 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:47.549065    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:47.698314 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:47.697472    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:49.572387 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:49.572353    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:50.898170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:50.898134    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:51.076969 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:51.076932    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:51.433324 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:51.433292    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:52.761512 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.761464    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:52.903209 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.903171    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"]
Nov 02 12:54:52.903971 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.903943    5879 cpu_manager.go:414] "containerMap: removing container" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy"
Nov 02 12:54:52.903971 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.903967    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy"
Nov 02 12:54:52.904196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.904128    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b970dd-6fcd-4e7a-913a-03abc88d0969" containerName="istio-proxy"
Nov 02 12:54:52.907715 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.907696    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"
Nov 02 12:54:52.917663 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.917632    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:52.924695 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.924674    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-43be82-dockercfg-wwr9t\""
Nov 02 12:54:52.930569 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.930541    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"]
Nov 02 12:54:52.937904 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.937874    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-43be82\""
Nov 02 12:54:52.958213 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.958176    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47674527-cd9d-480e-b292-776c382c168d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf\" (UID: \"47674527-cd9d-480e-b292-776c382c168d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"
Nov 02 12:54:52.958408 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.958319    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpr25\" (UniqueName: \"kubernetes.io/projected/47674527-cd9d-480e-b292-776c382c168d-kube-api-access-lpr25\") pod \"payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf\" (UID: \"47674527-cd9d-480e-b292-776c382c168d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"
Nov 02 12:54:52.999654 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:52.999621    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"]
Nov 02 12:54:53.004078 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.004054    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"
Nov 02 12:54:53.069017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.067432    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"]
Nov 02 12:54:53.090705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.090669    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpr25\" (UniqueName: \"kubernetes.io/projected/47674527-cd9d-480e-b292-776c382c168d-kube-api-access-lpr25\") pod \"payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf\" (UID: \"47674527-cd9d-480e-b292-776c382c168d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"
Nov 02 12:54:53.091210 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.091186    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47674527-cd9d-480e-b292-776c382c168d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf\" (UID: \"47674527-cd9d-480e-b292-776c382c168d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"
Nov 02 12:54:53.092228 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.092204    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47674527-cd9d-480e-b292-776c382c168d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf\" (UID: \"47674527-cd9d-480e-b292-776c382c168d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"
Nov 02 12:54:53.185036 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.171428    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpr25\" (UniqueName: \"kubernetes.io/projected/47674527-cd9d-480e-b292-776c382c168d-kube-api-access-lpr25\") pod \"payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf\" (UID: \"47674527-cd9d-480e-b292-776c382c168d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"
Nov 02 12:54:53.203857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.203815    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbjxn\" (UniqueName: \"kubernetes.io/projected/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-kube-api-access-dbjxn\") pod \"payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh\" (UID: \"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"
Nov 02 12:54:53.204106 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.203924    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh\" (UID: \"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"
Nov 02 12:54:53.223256 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.223210    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"
Nov 02 12:54:53.307170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.307080    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbjxn\" (UniqueName: \"kubernetes.io/projected/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-kube-api-access-dbjxn\") pod \"payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh\" (UID: \"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"
Nov 02 12:54:53.307170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.307151    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh\" (UID: \"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"
Nov 02 12:54:53.308483 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.308455    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh\" (UID: \"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"
Nov 02 12:54:53.343500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.343472    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbjxn\" (UniqueName: \"kubernetes.io/projected/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-kube-api-access-dbjxn\") pod \"payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh\" (UID: \"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"
Nov 02 12:54:53.389851 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.389822    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"
Nov 02 12:54:53.552789 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:54:53.552761    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47674527_cd9d_480e_b292_776c382c168d.slice/crio-979cd7561738a7b6d565467320e5da0c0c0bcb2507ae188e41d1c505033e8f31 WatchSource:0}: Error finding container 979cd7561738a7b6d565467320e5da0c0c0bcb2507ae188e41d1c505033e8f31: Status 404 returned error can't find the container with id 979cd7561738a7b6d565467320e5da0c0c0bcb2507ae188e41d1c505033e8f31
Nov 02 12:54:53.555947 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.555835    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"]
Nov 02 12:54:53.721059 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:54:53.721027    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99cd497_48ae_4d1d_adc8_4c4b7ab018f3.slice/crio-ec3a852fafe0e4e39b4671a4d44127eb068e0cbd2baf52be7dc748b6f570aa0a WatchSource:0}: Error finding container ec3a852fafe0e4e39b4671a4d44127eb068e0cbd2baf52be7dc748b6f570aa0a: Status 404 returned error can't find the container with id ec3a852fafe0e4e39b4671a4d44127eb068e0cbd2baf52be7dc748b6f570aa0a
Nov 02 12:54:53.723959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.723931    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"]
Nov 02 12:54:53.811806 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.811765    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:53.999233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.999203    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf" event={"ID":"47674527-cd9d-480e-b292-776c382c168d","Type":"ContainerStarted","Data":"869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1"}
Nov 02 12:54:53.999233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.999251    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf" event={"ID":"47674527-cd9d-480e-b292-776c382c168d","Type":"ContainerStarted","Data":"979cd7561738a7b6d565467320e5da0c0c0bcb2507ae188e41d1c505033e8f31"}
Nov 02 12:54:53.999486 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:53.999311    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"
Nov 02 12:54:54.001024 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:54.000981    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh" event={"ID":"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3","Type":"ContainerStarted","Data":"fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c"}
Nov 02 12:54:54.001138 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:54.001055    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"
Nov 02 12:54:54.001138 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:54.001078    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh" event={"ID":"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3","Type":"ContainerStarted","Data":"ec3a852fafe0e4e39b4671a4d44127eb068e0cbd2baf52be7dc748b6f570aa0a"}
Nov 02 12:54:54.058348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:54.058290    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf" podStartSLOduration=2.058273658 podStartE2EDuration="2.058273658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:54.050967676 +0000 UTC m=+1640.157670530" watchObservedRunningTime="2026-11-02 12:54:54.058273658 +0000 UTC m=+1640.164976508"
Nov 02 12:54:54.136726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:54.136669    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh" podStartSLOduration=2.136656516 podStartE2EDuration="2.136656516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:54:54.132362695 +0000 UTC m=+1640.239065542" watchObservedRunningTime="2026-11-02 12:54:54.136656516 +0000 UTC m=+1640.243359366"
Nov 02 12:54:55.534126 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:55.534090    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:57.378451 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:57.378401    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:54:59.255424 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:54:59.242180    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:01.012348 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:01.011651    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:02.697360 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:02.693071    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:04.545493 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:04.545461    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:05.012895 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:05.012860    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"
Nov 02 12:55:05.013144 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:05.012928    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"
Nov 02 12:55:05.300120 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:05.300040    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:06.164907 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:06.164864    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:07.850057 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:07.850024    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:08.457038 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.453282    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"]
Nov 02 12:55:08.460023 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.459964    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.488501 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.488471    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"]
Nov 02 12:55:08.512205 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.512162    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-c0d4cb\"/\"kube-root-ca.crt\""
Nov 02 12:55:08.512409 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.512292    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-c0d4cb\"/\"openshift-service-ca.crt\""
Nov 02 12:55:08.512503 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.512407    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-c0d4cb\"/\"praxis-route-c0d4cb-kserve-self-signed-certs\""
Nov 02 12:55:08.512575 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.512539    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-c0d4cb\"/\"default-dockercfg-d7zjq\""
Nov 02 12:55:08.537953 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.537909    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8mx\" (UniqueName: \"kubernetes.io/projected/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-kube-api-access-lv8mx\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.538170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.537970    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-dshm\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.538170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.538105    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-model-cache\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.538170 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.538159    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tmp-dir\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.538302 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.538192    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tls-certs\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.538363 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.538342    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-home\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.590879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.590845    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:08.640480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.640443    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8mx\" (UniqueName: \"kubernetes.io/projected/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-kube-api-access-lv8mx\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.640680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.640488    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-dshm\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.640680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.640517    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-model-cache\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.640680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.640553    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tmp-dir\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.640680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.640583    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tls-certs\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.640680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.640678    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-home\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.641049 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.641024    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-model-cache\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.641513 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.641483    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tmp-dir\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.641672 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.641483    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-home\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.643207 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.643180    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-dshm\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.643353 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.643334    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tls-certs\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.665769 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.665735    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8mx\" (UniqueName: \"kubernetes.io/projected/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-kube-api-access-lv8mx\") pod \"praxis-route-c0d4cb-kserve-6998675845-czkll\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") " pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:08.738880 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.738694    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:08.792024 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:08.788337    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:09.159522 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:09.159499    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"]
Nov 02 12:55:09.162454 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:55:09.162422    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd735452_8ab9_4aa8_ab53_a83cc93f7ae9.slice/crio-4c172bd1252aaa218be004655c25c580ace7b8d7e7162c6d858b3855089d5dec WatchSource:0}: Error finding container 4c172bd1252aaa218be004655c25c580ace7b8d7e7162c6d858b3855089d5dec: Status 404 returned error can't find the container with id 4c172bd1252aaa218be004655c25c580ace7b8d7e7162c6d858b3855089d5dec
Nov 02 12:55:09.978763 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:09.978731    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:10.130007 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:10.129445    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll" event={"ID":"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9","Type":"ContainerStarted","Data":"42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576"}
Nov 02 12:55:10.130007 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:10.129501    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll" event={"ID":"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9","Type":"ContainerStarted","Data":"4c172bd1252aaa218be004655c25c580ace7b8d7e7162c6d858b3855089d5dec"}
Nov 02 12:55:10.179603 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:10.179550    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll" podStartSLOduration=2.179535243 podStartE2EDuration="2.179535243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:55:10.178746066 +0000 UTC m=+1656.285448919" watchObservedRunningTime="2026-11-02 12:55:10.179535243 +0000 UTC m=+1656.286238093"
Nov 02 12:55:10.789232 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:10.789202    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:11.900489 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:11.900452    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:13.876484 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:13.876446    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:14.463765 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:14.463730    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:14.822064 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:55:14.821966    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:55:14.993683 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:14.993621    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:15.150659 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:15.150627    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:15.754842 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:15.754802    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:17.365548 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:17.365518    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:17.561644 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:17.561614    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:17.708011 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:17.707905    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:18.789535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:18.789498    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:18.802981 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:18.802958    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:19.214844 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:19.214819    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:55:19.277145 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:19.277111    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:20.867714 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:20.867675    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:21.616912 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:21.616841    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:22.647892 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:22.647863    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:24.257667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:24.257632    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:25.866262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:25.866230    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:27.610237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:27.610205    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:29.244799 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:29.244764    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:30.851442 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.851413    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"]
Nov 02 12:55:30.855935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.855909    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:30.878546 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.878522    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"]
Nov 02 12:55:30.885922 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.885894    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a6f2c0\"/\"openshift-service-ca.crt\""
Nov 02 12:55:30.886086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.886018    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a6f2c0\"/\"auth-test-model-a6f2c0-kserve-self-signed-certs\""
Nov 02 12:55:30.886086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.886037    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a6f2c0\"/\"kube-root-ca.crt\""
Nov 02 12:55:30.886086 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.886064    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a6f2c0\"/\"default-dockercfg-vf8k9\""
Nov 02 12:55:30.927724 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.927700    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b47ed8-a443-4f92-9589-3288fb61d71d-tls-certs\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:30.927858 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.927772    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-model-cache\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:30.927902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.927856    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-home\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:30.928031 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.927970    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-dshm\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:30.928112 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.928077    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8wn\" (UniqueName: \"kubernetes.io/projected/f5b47ed8-a443-4f92-9589-3288fb61d71d-kube-api-access-zc8wn\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:30.928215 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.928203    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-tmp-dir\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:30.966337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:30.966312    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:31.029610 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.029582    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-model-cache\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.029769 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.029635    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-home\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.029769 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.029718    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-dshm\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.029900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.029771    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8wn\" (UniqueName: \"kubernetes.io/projected/f5b47ed8-a443-4f92-9589-3288fb61d71d-kube-api-access-zc8wn\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.029900 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.029893    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-tmp-dir\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.030033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.029962    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b47ed8-a443-4f92-9589-3288fb61d71d-tls-certs\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.030033 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.030012    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-model-cache\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.030157 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.030061    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-home\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.030270 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.030253    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-tmp-dir\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.032178 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.032150    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-dshm\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.032706 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.032686    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b47ed8-a443-4f92-9589-3288fb61d71d-tls-certs\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.061605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.061580    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8wn\" (UniqueName: \"kubernetes.io/projected/f5b47ed8-a443-4f92-9589-3288fb61d71d-kube-api-access-zc8wn\") pod \"auth-test-model-a6f2c0-kserve-7d9864f797-87dzz\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.130908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.130858    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:31.176227 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.176197    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:31.562752 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:55:31.562722    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5b47ed8_a443_4f92_9589_3288fb61d71d.slice/crio-b69870e1be0d3ce3468d5da982acdc7a4abbdda5141fc67b073e3917abd371ae WatchSource:0}: Error finding container b69870e1be0d3ce3468d5da982acdc7a4abbdda5141fc67b073e3917abd371ae: Status 404 returned error can't find the container with id b69870e1be0d3ce3468d5da982acdc7a4abbdda5141fc67b073e3917abd371ae
Nov 02 12:55:31.564549 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:31.564529    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"]
Nov 02 12:55:32.332081 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:32.332043    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz" event={"ID":"f5b47ed8-a443-4f92-9589-3288fb61d71d","Type":"ContainerStarted","Data":"cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c"}
Nov 02 12:55:32.332465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:32.332090    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz" event={"ID":"f5b47ed8-a443-4f92-9589-3288fb61d71d","Type":"ContainerStarted","Data":"b69870e1be0d3ce3468d5da982acdc7a4abbdda5141fc67b073e3917abd371ae"}
Nov 02 12:55:32.381514 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:32.381452    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz" podStartSLOduration=2.381433824 podStartE2EDuration="2.381433824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:55:32.375981881 +0000 UTC m=+1678.482684733" watchObservedRunningTime="2026-11-02 12:55:32.381433824 +0000 UTC m=+1678.488136677"
Nov 02 12:55:32.671849 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:32.671819    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:33.176506 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.176474    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:33.210494 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.210437    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"]
Nov 02 12:55:33.216977 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.216945    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.235867 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.235849    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"]
Nov 02 12:55:33.245112 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.245067    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-ed56b9\"/\"default-dockercfg-fvspn\""
Nov 02 12:55:33.245262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.245121    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-ed56b9\"/\"openshift-service-ca.crt\""
Nov 02 12:55:33.245262 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.245255    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-ed56b9\"/\"auth-test-model-ed56b9-kserve-self-signed-certs\""
Nov 02 12:55:33.247299 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.247283    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-ed56b9\"/\"kube-root-ca.crt\""
Nov 02 12:55:33.339979 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.339949    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:33.375457 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.375428    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-home\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.375641 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.375560    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-model-cache\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.375641 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.375616    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pws6m\" (UniqueName: \"kubernetes.io/projected/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-kube-api-access-pws6m\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.375758 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.375659    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tmp-dir\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.375858 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.375836    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tls-certs\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.375902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.375876    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-dshm\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.483064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.481084    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tls-certs\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.483064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.481138    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-dshm\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.483064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.481366    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-home\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.483064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.481467    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-model-cache\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.483064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.481544    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pws6m\" (UniqueName: \"kubernetes.io/projected/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-kube-api-access-pws6m\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.483064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.481600    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tmp-dir\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.483064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.482145    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tmp-dir\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.483064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.482414    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-home\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.483640 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.483263    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-model-cache\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.513016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.510639    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tls-certs\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.513016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.510920    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-dshm\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.523045 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.521806    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pws6m\" (UniqueName: \"kubernetes.io/projected/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-kube-api-access-pws6m\") pod \"auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.546016 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.539927    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:33.926503 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:55:33.926469    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1176222b_0fa2_40d2_898b_9a6fcd8d35aa.slice/crio-d9fd3548ac861667a66023c5d3e0398fc165da3b1f9e1e12e492a606b98e43ca WatchSource:0}: Error finding container d9fd3548ac861667a66023c5d3e0398fc165da3b1f9e1e12e492a606b98e43ca: Status 404 returned error can't find the container with id d9fd3548ac861667a66023c5d3e0398fc165da3b1f9e1e12e492a606b98e43ca
Nov 02 12:55:33.926917 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:33.926898    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"]
Nov 02 12:55:34.356706 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:34.356622    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t" event={"ID":"1176222b-0fa2-40d2-898b-9a6fcd8d35aa","Type":"ContainerStarted","Data":"6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3"}
Nov 02 12:55:34.356706 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:34.356676    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t" event={"ID":"1176222b-0fa2-40d2-898b-9a6fcd8d35aa","Type":"ContainerStarted","Data":"d9fd3548ac861667a66023c5d3e0398fc165da3b1f9e1e12e492a606b98e43ca"}
Nov 02 12:55:34.393717 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:34.393682    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:34.406715 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:34.406649    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t" podStartSLOduration=1.406630175 podStartE2EDuration="1.406630175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:55:34.403529666 +0000 UTC m=+1680.510232518" watchObservedRunningTime="2026-11-02 12:55:34.406630175 +0000 UTC m=+1680.513333031"
Nov 02 12:55:34.540398 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:34.540327    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:34.695195 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:34.695157    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:35.958615 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:35.958580    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:36.555569 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:36.555539    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:37.619367 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:37.619337    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:39.151440 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:39.151407    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:40.822281 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.822247    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:40.897789 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.897754    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"]
Nov 02 12:55:40.904461 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.904438    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"]
Nov 02 12:55:40.904572 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.904540    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:40.924862 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.924836    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e7561a\"/\"openshift-service-ca.crt\""
Nov 02 12:55:40.925017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.924916    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e7561a\"/\"kube-root-ca.crt\""
Nov 02 12:55:40.944055 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.938363    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e7561a\"/\"default-dockercfg-cwcd9\""
Nov 02 12:55:40.944055 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.938414    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e7561a\"/\"auth-test-model-e7561a-kserve-self-signed-certs\""
Nov 02 12:55:40.960688 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.960661    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:40.999216 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.999186    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-model-cache\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:40.999518 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.999496    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-dshm\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:40.999665 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.999648    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-home\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:40.999822 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.999806    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txgm6\" (UniqueName: \"kubernetes.io/projected/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-kube-api-access-txgm6\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:40.999949 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:40.999931    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tmp-dir\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.000260 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.000242    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tls-certs\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.101221 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.101146    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txgm6\" (UniqueName: \"kubernetes.io/projected/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-kube-api-access-txgm6\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.101221 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.101183    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tmp-dir\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.101432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.101252    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tls-certs\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.101432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.101331    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-model-cache\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.101432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.101396    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-dshm\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.101432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.101425    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-home\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.101674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.101653    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tmp-dir\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.101740 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.101721    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-model-cache\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.102085 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.102058    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-home\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.103888 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.103863    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-dshm\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.104014 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.103981    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tls-certs\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.134796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.134734    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txgm6\" (UniqueName: \"kubernetes.io/projected/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-kube-api-access-txgm6\") pod \"auth-test-model-e7561a-kserve-6bc7696db5-fk7k7\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") " pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.181543 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.180618    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:41.222533 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.222269    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:41.256511 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.256095    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:41.420785 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.420754    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 12:55:41.563649 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:55:41.563620    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd83a9abe_0c3c_44b2_8961_378aa63ce5ea.slice/crio-46acc46ea78aa6c546189ed0aff17043a6e600475497ba84fc309200dc24efe1 WatchSource:0}: Error finding container 46acc46ea78aa6c546189ed0aff17043a6e600475497ba84fc309200dc24efe1: Status 404 returned error can't find the container with id 46acc46ea78aa6c546189ed0aff17043a6e600475497ba84fc309200dc24efe1
Nov 02 12:55:41.564894 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:41.564870    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"]
Nov 02 12:55:42.405391 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:42.405353    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:42.418838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:42.418803    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7" event={"ID":"d83a9abe-0c3c-44b2-8961-378aa63ce5ea","Type":"ContainerStarted","Data":"cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c"}
Nov 02 12:55:42.419059 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:42.418862    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7" event={"ID":"d83a9abe-0c3c-44b2-8961-378aa63ce5ea","Type":"ContainerStarted","Data":"46acc46ea78aa6c546189ed0aff17043a6e600475497ba84fc309200dc24efe1"}
Nov 02 12:55:42.472624 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:42.472558    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7" podStartSLOduration=2.472537639 podStartE2EDuration="2.472537639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:55:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:55:42.467179372 +0000 UTC m=+1688.573882225" watchObservedRunningTime="2026-11-02 12:55:42.472537639 +0000 UTC m=+1688.579240492"
Nov 02 12:55:43.223049 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:43.223015    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:43.540604 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:43.540530    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:43.553687 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:43.553664    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:43.939867 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:43.939833    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:44.445627 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:44.445597    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 12:55:44.823248 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:55:44.823145    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:55:45.502726 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:45.502692    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:46.929668 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:46.929636    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:47.310728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:47.310634    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:48.876701 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:48.876666    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:50.468645 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:50.468593    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:51.223414 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:51.223383    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:51.237079 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:51.237057    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:51.528083 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:51.528005    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 12:55:52.143323 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:52.143286    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:53.956053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:53.956018    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:55.584724 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:55.584380    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:56.794951 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:56.794912    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:57.248867 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:57.248833    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:55:58.888253 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:55:58.888219    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:00.493237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:00.493206    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:01.033057 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:01.033016    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"]
Nov 02 12:56:01.046200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:01.046170    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43636316-twlzl"]
Nov 02 12:56:02.018162 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:02.018133    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7965e746-da50-44ab-aa60-67505ec31da8" path="/var/lib/kubelet/pods/7965e746-da50-44ab-aa60-67505ec31da8/volumes"
Nov 02 12:56:02.067708 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:02.067676    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:03.659152 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:03.659121    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:05.360928 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:05.360875    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:06.846284 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:06.846251    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:08.454166 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:08.454129    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:09.952405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:09.952374    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:11.661087 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:11.661054    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:13.363624 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:13.363593    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:14.820793 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:56:14.820752    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:56:15.253443 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:15.253407    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:16.654591 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:16.654555    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:17.032383 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:17.032284    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"]
Nov 02 12:56:17.046291 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:17.046264    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9cb73b3-69qgl"]
Nov 02 12:56:18.018381 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:18.018350    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3500de69-4911-4539-a6a9-48830aa2a6bf" path="/var/lib/kubelet/pods/3500de69-4911-4539-a6a9-48830aa2a6bf/volumes"
Nov 02 12:56:18.338577 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:18.338500    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:20.056128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.056096    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:20.394734 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.394700    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"]
Nov 02 12:56:20.401244 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.401218    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.424451 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.424425    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"]
Nov 02 12:56:20.432432 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.432406    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-1aa18c\"/\"default-dockercfg-2xjds\""
Nov 02 12:56:20.432611 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.432441    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-1aa18c\"/\"kube-root-ca.crt\""
Nov 02 12:56:20.432611 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.432464    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-1aa18c\"/\"auth-test-model-1aa18c-kserve-self-signed-certs\""
Nov 02 12:56:20.432795 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.432520    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-1aa18c\"/\"openshift-service-ca.crt\""
Nov 02 12:56:20.473108 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.473065    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:20.545500 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.545466    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-dshm\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.545662 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.545520    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvq5j\" (UniqueName: \"kubernetes.io/projected/70bda333-74a7-4dad-8e23-7c98eb74e540-kube-api-access-mvq5j\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.545709 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.545675    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bda333-74a7-4dad-8e23-7c98eb74e540-tls-certs\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.545816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.545798    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-tmp-dir\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.545861 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.545825    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-home\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.545936 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.545925    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-model-cache\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.649843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.648470    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-tmp-dir\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.649843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.648538    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-home\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.649843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.648677    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-model-cache\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.649843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.648840    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-dshm\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.649843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.648885    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvq5j\" (UniqueName: \"kubernetes.io/projected/70bda333-74a7-4dad-8e23-7c98eb74e540-kube-api-access-mvq5j\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.649843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.649021    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bda333-74a7-4dad-8e23-7c98eb74e540-tls-certs\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.670018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.655417    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-model-cache\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.670018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.655664    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-tmp-dir\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.670018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.655883    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-home\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.670018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.659609    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-dshm\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.712878 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.698046    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bda333-74a7-4dad-8e23-7c98eb74e540-tls-certs\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.712878 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.703602    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvq5j\" (UniqueName: \"kubernetes.io/projected/70bda333-74a7-4dad-8e23-7c98eb74e540-kube-api-access-mvq5j\") pod \"auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:20.724765 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:20.724736    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:21.112852 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:56:21.112822    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70bda333_74a7_4dad_8e23_7c98eb74e540.slice/crio-b81fcdb532542c1c207f63f9cefba29c986ff7237a452879324e1c9a691651a6 WatchSource:0}: Error finding container b81fcdb532542c1c207f63f9cefba29c986ff7237a452879324e1c9a691651a6: Status 404 returned error can't find the container with id b81fcdb532542c1c207f63f9cefba29c986ff7237a452879324e1c9a691651a6
Nov 02 12:56:21.114543 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:21.114522    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"]
Nov 02 12:56:21.787409 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:21.787368    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:21.792189 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:21.792151    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99" event={"ID":"70bda333-74a7-4dad-8e23-7c98eb74e540","Type":"ContainerStarted","Data":"79c52a6f15e05346b9f6bb3d9e051b38b586b7e2a2f2d2dec0fbc9a9ee26e059"}
Nov 02 12:56:21.792683 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:21.792655    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99" event={"ID":"70bda333-74a7-4dad-8e23-7c98eb74e540","Type":"ContainerStarted","Data":"b81fcdb532542c1c207f63f9cefba29c986ff7237a452879324e1c9a691651a6"}
Nov 02 12:56:21.839059 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:21.838979    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99" podStartSLOduration=1.8389593560000002 podStartE2EDuration="1.838959356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:56:21.837563874 +0000 UTC m=+1727.944266725" watchObservedRunningTime="2026-11-02 12:56:21.838959356 +0000 UTC m=+1727.945662211"
Nov 02 12:56:22.593833 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:22.593779    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:22.732085 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:22.726072    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:23.368233 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:23.368195    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:24.536590 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:24.536554    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:25.173966 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:25.173839    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:26.795416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:26.795383    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:28.273072 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:28.273038    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:29.851799 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:29.851766    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:30.726397 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:30.726358    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:30.739201 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:30.739169    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:30.927693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:30.927665    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 12:56:31.522816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:31.522783    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:33.036060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:33.036024    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"]
Nov 02 12:56:33.051284 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:33.051241    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f9a5c5-c6f2n"]
Nov 02 12:56:33.238643 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:33.238612    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:34.018290 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:34.018256    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912751e5-8e00-41f4-b45e-6a8244add550" path="/var/lib/kubelet/pods/912751e5-8e00-41f4-b45e-6a8244add550/volumes"
Nov 02 12:56:34.775358 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:34.775316    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:34.902784 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:34.902752    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:36.368717 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:36.368684    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:36.683749 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:36.683673    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:37.966533 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:37.966487    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:39.780605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:39.780490    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:41.357742 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:41.357710    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:41.493882 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:41.493782    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:43.095681 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:43.095625    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:44.613096 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:44.613054    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:44.823298 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:56:44.823265    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:56:46.227594 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:46.227560    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:46.322174 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:46.322142    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"]
Nov 02 12:56:46.433638 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:46.433602    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:47.034931 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:47.034897    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"]
Nov 02 12:56:47.048454 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:47.048422    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aecfd915-ggxzz"]
Nov 02 12:56:47.821888 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:47.821852    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:48.018696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:48.018657    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac" path="/var/lib/kubelet/pods/ae13d44e-ca74-4eb8-8bd8-3d9c1ee852ac/volumes"
Nov 02 12:56:49.396044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:49.395052    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:50.973350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:50.973315    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:52.550650 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:52.550610    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:54.200636 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:54.200603    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:55.043422 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:55.043389    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"]
Nov 02 12:56:55.056440 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:55.056410    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42499d2f-24bsz"]
Nov 02 12:56:55.738272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:55.738232    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:56.019357 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:56.019273    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b9b75f-b1e5-4e8c-9989-c4d3059422d2" path="/var/lib/kubelet/pods/25b9b75f-b1e5-4e8c-9989-c4d3059422d2/volumes"
Nov 02 12:56:56.032558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:56.032522    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"]
Nov 02 12:56:56.046177 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:56.046144    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefe46e5-ftgfh"]
Nov 02 12:56:57.033720 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:57.033682    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"]
Nov 02 12:56:57.049064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:57.049030    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a532c0-lfvvd"]
Nov 02 12:56:57.414969 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:57.414933    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:57.663396 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:57.663360    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:56:58.017813 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:58.017778    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b7068f-e026-4857-bac2-bd487fca458e" path="/var/lib/kubelet/pods/07b7068f-e026-4857-bac2-bd487fca458e/volumes"
Nov 02 12:56:58.018328 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:58.018313    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f926c748-f9e9-4be9-9ae4-a06c61a76631" path="/var/lib/kubelet/pods/f926c748-f9e9-4be9-9ae4-a06c61a76631/volumes"
Nov 02 12:56:59.017196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:56:59.017165    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:00.613603 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:00.613553    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:01.347090 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.346974    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll" podUID="fd735452-8ab9-4aa8-ab53-a83cc93f7ae9" containerName="main" containerID="cri-o://42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576" gracePeriod=45
Nov 02 12:57:01.640441 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.640411    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:57:01.829818 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.829783    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-model-cache\") pod \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") "
Nov 02 12:57:01.830052 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.829840    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-dshm\") pod \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") "
Nov 02 12:57:01.830052 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.829879    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tls-certs\") pod \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") "
Nov 02 12:57:01.830052 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.829931    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-kube-api-access-lv8mx\" (UniqueName: \"kubernetes.io/projected/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-kube-api-access-lv8mx\") pod \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") "
Nov 02 12:57:01.830264 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.830122    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-model-cache" pod "fd735452-8ab9-4aa8-ab53-a83cc93f7ae9" (UID: "fd735452-8ab9-4aa8-ab53-a83cc93f7ae9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:57:01.830264 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.830241    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tmp-dir\") pod \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") "
Nov 02 12:57:01.830374 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.830292    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-home\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-home\") pod \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\" (UID: \"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9\") "
Nov 02 12:57:01.830533 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.830510    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tmp-dir" pod "fd735452-8ab9-4aa8-ab53-a83cc93f7ae9" (UID: "fd735452-8ab9-4aa8-ab53-a83cc93f7ae9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:57:01.830865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.830796    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-home" pod "fd735452-8ab9-4aa8-ab53-a83cc93f7ae9" (UID: "fd735452-8ab9-4aa8-ab53-a83cc93f7ae9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:57:01.831043 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.830986    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:57:01.831127 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.831050    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:57:01.831127 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.831065    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:57:01.832371 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.832349    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-dshm" pod "fd735452-8ab9-4aa8-ab53-a83cc93f7ae9" (UID: "fd735452-8ab9-4aa8-ab53-a83cc93f7ae9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:57:01.832506 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.832451    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-kube-api-access-lv8mx" pod "fd735452-8ab9-4aa8-ab53-a83cc93f7ae9" (UID: "fd735452-8ab9-4aa8-ab53-a83cc93f7ae9"). InnerVolumeSpecName "kube-api-access-lv8mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:57:01.832506 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.832492    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tls-certs" pod "fd735452-8ab9-4aa8-ab53-a83cc93f7ae9" (UID: "fd735452-8ab9-4aa8-ab53-a83cc93f7ae9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:57:01.932712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.932679    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv8mx\" (UniqueName: \"kubernetes.io/projected/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-kube-api-access-lv8mx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:57:01.932712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.932711    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:57:01.932933 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:01.932722    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:57:02.157889 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:02.157853    5879 generic.go:356] "Generic (PLEG): container finished" podID="fd735452-8ab9-4aa8-ab53-a83cc93f7ae9" containerID="42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576" exitCode=0
Nov 02 12:57:02.157889 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:02.157893    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"
Nov 02 12:57:02.158128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:02.157929    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll" event={"ID":"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9","Type":"ContainerDied","Data":"42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576"}
Nov 02 12:57:02.158128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:02.157969    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll" event={"ID":"fd735452-8ab9-4aa8-ab53-a83cc93f7ae9","Type":"ContainerDied","Data":"4c172bd1252aaa218be004655c25c580ace7b8d7e7162c6d858b3855089d5dec"}
Nov 02 12:57:02.158128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:02.157986    5879 scope.go:122] "RemoveContainer" containerID="42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576"
Nov 02 12:57:02.169099 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:02.169062    5879 scope.go:122] "RemoveContainer" containerID="42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576"
Nov 02 12:57:02.169407 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:57:02.169385    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576\": container with ID starting with 42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576 not found: ID does not exist" containerID="42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576"
Nov 02 12:57:02.169461 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:02.169416    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576"} err="failed to get container status \"42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576\": rpc error: code = NotFound desc = could not find container \"42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576\": container with ID starting with 42ec5b7eaed9809b9833cd58865ea00ae2a83d56dfac15a1211e05aa2c22e576 not found: ID does not exist"
Nov 02 12:57:02.224075 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:02.223982    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"]
Nov 02 12:57:02.227711 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:02.227682    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-c0d4cb/praxis-route-c0d4cb-kserve-6998675845-czkll"]
Nov 02 12:57:02.302729 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:02.302694    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:03.203222 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.203188    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"]
Nov 02 12:57:03.203950 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.203934    5879 cpu_manager.go:414] "containerMap: removing container" podUID="fd735452-8ab9-4aa8-ab53-a83cc93f7ae9" containerName="main"
Nov 02 12:57:03.204032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.203952    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd735452-8ab9-4aa8-ab53-a83cc93f7ae9" containerName="main"
Nov 02 12:57:03.204126 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.204115    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd735452-8ab9-4aa8-ab53-a83cc93f7ae9" containerName="main"
Nov 02 12:57:03.209353 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.209330    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:03.228102 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.228076    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"]
Nov 02 12:57:03.236514 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.236474    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:57:03.236770 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.236754    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:57:03.252343 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.252311    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae37406-kxmx2\" (UID: \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:03.252491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.252393    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae37406-kxmx2\" (UID: \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:03.252677 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.252652    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45vkn\" (UniqueName: \"kubernetes.io/projected/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-kube-api-access-45vkn\") pod \"maas-api-revoke-sub-3ae37406-kxmx2\" (UID: \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:03.353508 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.353473    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae37406-kxmx2\" (UID: \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:03.353508 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.353509    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae37406-kxmx2\" (UID: \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:03.353779 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.353672    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45vkn\" (UniqueName: \"kubernetes.io/projected/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-kube-api-access-45vkn\") pod \"maas-api-revoke-sub-3ae37406-kxmx2\" (UID: \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:03.354275 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.354252    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ae37406-kxmx2\" (UID: \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:03.356323 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.356302    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ae37406-kxmx2\" (UID: \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:03.378917 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.378882    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45vkn\" (UniqueName: \"kubernetes.io/projected/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-kube-api-access-45vkn\") pod \"maas-api-revoke-sub-3ae37406-kxmx2\" (UID: \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:03.524068 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.523966    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:03.837612 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:57:03.837278    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4fdad1_9361_4f4e_8f4d_d68dae46d602.slice/crio-4a9e743d6f0ee93ab88ec6e61b828122071ae756034d2505650879851b134f04 WatchSource:0}: Error finding container 4a9e743d6f0ee93ab88ec6e61b828122071ae756034d2505650879851b134f04: Status 404 returned error can't find the container with id 4a9e743d6f0ee93ab88ec6e61b828122071ae756034d2505650879851b134f04
Nov 02 12:57:03.838340 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.838317    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"]
Nov 02 12:57:03.885893 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:03.885861    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:04.025954 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:04.025244    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd735452-8ab9-4aa8-ab53-a83cc93f7ae9" path="/var/lib/kubelet/pods/fd735452-8ab9-4aa8-ab53-a83cc93f7ae9/volumes"
Nov 02 12:57:04.098057 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:04.097947    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:04.181355 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:04.180863    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2" event={"ID":"9d4fdad1-9361-4f4e-8f4d-d68dae46d602","Type":"ContainerStarted","Data":"4a9e743d6f0ee93ab88ec6e61b828122071ae756034d2505650879851b134f04"}
Nov 02 12:57:04.405425 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:04.405391    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:04.606722 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:04.606681    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:04.746583 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:04.746501    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:05.192385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:05.192349    5879 generic.go:356] "Generic (PLEG): container finished" podID="9d4fdad1-9361-4f4e-8f4d-d68dae46d602" containerID="a26f63870921a4512a8e0d1dfc24a795e35a7fa8bca0c24afcba3c9ff6277f0b" exitCode=0
Nov 02 12:57:05.192546 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:05.192433    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2" event={"ID":"9d4fdad1-9361-4f4e-8f4d-d68dae46d602","Type":"ContainerDied","Data":"a26f63870921a4512a8e0d1dfc24a795e35a7fa8bca0c24afcba3c9ff6277f0b"}
Nov 02 12:57:05.853543 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:05.853511    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:06.357861 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:06.357836    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:06.406812 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:06.406771    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-openshift-ca\") pod \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\" (UID: \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\") "
Nov 02 12:57:06.407041 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:06.406940    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-kube-api-access-45vkn\" (UniqueName: \"kubernetes.io/projected/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-kube-api-access-45vkn\") pod \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\" (UID: \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\") "
Nov 02 12:57:06.407202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:06.407180    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-xks-ca\") pod \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\" (UID: \"9d4fdad1-9361-4f4e-8f4d-d68dae46d602\") "
Nov 02 12:57:06.407273 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:06.407204    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-openshift-ca" pod "9d4fdad1-9361-4f4e-8f4d-d68dae46d602" (UID: "9d4fdad1-9361-4f4e-8f4d-d68dae46d602"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:57:06.407782 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:06.407757    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:57:06.410245 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:06.410217    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-xks-ca" pod "9d4fdad1-9361-4f4e-8f4d-d68dae46d602" (UID: "9d4fdad1-9361-4f4e-8f4d-d68dae46d602"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:57:06.410728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:06.410232    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-kube-api-access-45vkn" pod "9d4fdad1-9361-4f4e-8f4d-d68dae46d602" (UID: "9d4fdad1-9361-4f4e-8f4d-d68dae46d602"). InnerVolumeSpecName "kube-api-access-45vkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:57:06.509511 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:06.509429    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:57:06.509511 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:06.509457    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45vkn\" (UniqueName: \"kubernetes.io/projected/9d4fdad1-9361-4f4e-8f4d-d68dae46d602-kube-api-access-45vkn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:57:06.696476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:06.696438    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:07.210163 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:07.210133    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"
Nov 02 12:57:07.210592 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:07.210160    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2" event={"ID":"9d4fdad1-9361-4f4e-8f4d-d68dae46d602","Type":"ContainerDied","Data":"4a9e743d6f0ee93ab88ec6e61b828122071ae756034d2505650879851b134f04"}
Nov 02 12:57:07.210592 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:07.210218    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a9e743d6f0ee93ab88ec6e61b828122071ae756034d2505650879851b134f04"
Nov 02 12:57:07.550588 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:07.550502    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:08.032490 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:08.032462    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"]
Nov 02 12:57:08.045959 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:08.045929    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59eac350-dljsz"]
Nov 02 12:57:09.275948 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:09.275834    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:10.018203 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:10.018168    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583f2743-034a-4154-9c77-aed8d9fe3e6f" path="/var/lib/kubelet/pods/583f2743-034a-4154-9c77-aed8d9fe3e6f/volumes"
Nov 02 12:57:10.069465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:10.069421    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"]
Nov 02 12:57:10.082842 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:10.082806    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"]
Nov 02 12:57:10.086475 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:10.086458    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efe51eb8-hzr7r"]
Nov 02 12:57:10.090060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:10.090036    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f182b01c-hdwtv"]
Nov 02 12:57:10.786448 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:10.786416    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:10.911685 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:10.911653    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:12.017444 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:12.017411    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58527ce8-efa6-4415-a874-5d9676f27e07" path="/var/lib/kubelet/pods/58527ce8-efa6-4415-a874-5d9676f27e07/volumes"
Nov 02 12:57:12.018029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:12.017988    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a126d6-15b8-4395-b94d-853bbee759b2" path="/var/lib/kubelet/pods/90a126d6-15b8-4395-b94d-853bbee759b2/volumes"
Nov 02 12:57:12.468042 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:12.466691    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:14.044225 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:14.044191    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:14.822244 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:57:14.822207    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:57:15.542196 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:15.542144    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:16.866416 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:16.866378    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:17.078210 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:17.078177    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:18.713754 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:18.713724    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:20.261491 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:20.261445    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:21.033009 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:21.032959    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"]
Nov 02 12:57:21.045520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:21.045460    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25d64ce5-n55hb"]
Nov 02 12:57:21.845975 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:21.845942    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:22.018605 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:22.018574    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcdb9655-b573-4310-82be-ca490663c074" path="/var/lib/kubelet/pods/fcdb9655-b573-4310-82be-ca490663c074/volumes"
Nov 02 12:57:23.383251 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:23.381399    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:24.865910 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:24.865881    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:26.031708 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:26.031676    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"]
Nov 02 12:57:26.045480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:26.045450    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-603b62b8-kjh95"]
Nov 02 12:57:26.048885 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:26.048861    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"]
Nov 02 12:57:26.052237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:26.052214    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2018b275-f8kmc"]
Nov 02 12:57:26.531202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:26.531174    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:28.018501 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:28.018470    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b06bdd-5931-4326-b010-f2b311cffe9b" path="/var/lib/kubelet/pods/63b06bdd-5931-4326-b010-f2b311cffe9b/volumes"
Nov 02 12:57:28.018943 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:28.018930    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2523a19-b10c-47f6-81cc-71a8082b2ae9" path="/var/lib/kubelet/pods/f2523a19-b10c-47f6-81cc-71a8082b2ae9/volumes"
Nov 02 12:57:28.220066 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:28.220030    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:28.769339 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:28.769306    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:28.892240 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:28.892201    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:29.679978 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:29.679938    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:30.382885 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:30.382847    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:31.806355 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:31.806293    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:31.942859 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:31.942827    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:32.484408 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:32.484371    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:32.909507 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:32.909468    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:33.695064 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:33.695026    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:33.840588 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:33.840544    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:35.232429 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:35.232395    5879 scope.go:122] "RemoveContainer" containerID="64512a357070b08796d0e2f320480da0b3aa1df6fa5d94a607fdffa24952f2c0"
Nov 02 12:57:35.243571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:35.243549    5879 scope.go:122] "RemoveContainer" containerID="cc385bc40750e18556acd15c7c7b01f372312899fa60f478fe3b037acf00b08b"
Nov 02 12:57:35.254300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:35.254275    5879 scope.go:122] "RemoveContainer" containerID="044d2dfa9cac178033e26cf6af9949be5167da46bd9ca01cbd96aa7c21eff9a6"
Nov 02 12:57:35.397253 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:35.397196    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:36.812838 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:36.812804    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:36.948504 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:36.948473    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:38.516186 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:38.516131    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:40.052716 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:40.052677    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"]
Nov 02 12:57:40.138455 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:40.138420    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:40.274418 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:40.274386    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:40.496674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:40.496643    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:41.893489 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:41.893432    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:42.030434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:42.030402    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"]
Nov 02 12:57:42.044074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:42.044044    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0114666b-2c28g"]
Nov 02 12:57:43.432455 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:43.432415    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:44.018611 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:44.018582    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08e746bd-3082-4aad-8fb1-960b4b8c66a0" path="/var/lib/kubelet/pods/08e746bd-3082-4aad-8fb1-960b4b8c66a0/volumes"
Nov 02 12:57:44.822563 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:57:44.822513    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:57:45.033800 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.033762    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:45.059128 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.059077    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj" podUID="99f9f86a-b60e-4b2b-8535-c28fd981e6e6" containerName="payload-processing" containerID="cri-o://873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57" gracePeriod=25
Nov 02 12:57:45.349633 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.349606    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"
Nov 02 12:57:45.502195 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.502159    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-plugins-config-volume\") pod \"99f9f86a-b60e-4b2b-8535-c28fd981e6e6\" (UID: \"99f9f86a-b60e-4b2b-8535-c28fd981e6e6\") "
Nov 02 12:57:45.502401 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.502252    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-kube-api-access-wbp86\" (UniqueName: \"kubernetes.io/projected/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-kube-api-access-wbp86\") pod \"99f9f86a-b60e-4b2b-8535-c28fd981e6e6\" (UID: \"99f9f86a-b60e-4b2b-8535-c28fd981e6e6\") "
Nov 02 12:57:45.502619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.502596    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-plugins-config-volume" pod "99f9f86a-b60e-4b2b-8535-c28fd981e6e6" (UID: "99f9f86a-b60e-4b2b-8535-c28fd981e6e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:57:45.502780 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.502763    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:57:45.504764 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.504741    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-kube-api-access-wbp86" pod "99f9f86a-b60e-4b2b-8535-c28fd981e6e6" (UID: "99f9f86a-b60e-4b2b-8535-c28fd981e6e6"). InnerVolumeSpecName "kube-api-access-wbp86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:57:45.526244 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.526208    5879 generic.go:356] "Generic (PLEG): container finished" podID="99f9f86a-b60e-4b2b-8535-c28fd981e6e6" containerID="873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57" exitCode=0
Nov 02 12:57:45.526465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.526263    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"
Nov 02 12:57:45.526465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.526291    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj" event={"ID":"99f9f86a-b60e-4b2b-8535-c28fd981e6e6","Type":"ContainerDied","Data":"873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57"}
Nov 02 12:57:45.526465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.526376    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj" event={"ID":"99f9f86a-b60e-4b2b-8535-c28fd981e6e6","Type":"ContainerDied","Data":"ce03e33abd4f7cec1abb2688a62cd3fce0fd551d99173c3b2c554979b8bd2aff"}
Nov 02 12:57:45.526465 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.526408    5879 scope.go:122] "RemoveContainer" containerID="873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57"
Nov 02 12:57:45.538237 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.538155    5879 scope.go:122] "RemoveContainer" containerID="873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57"
Nov 02 12:57:45.538497 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:57:45.538476    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57\": container with ID starting with 873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57 not found: ID does not exist" containerID="873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57"
Nov 02 12:57:45.538550 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.538506    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57"} err="failed to get container status \"873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57\": rpc error: code = NotFound desc = could not find container \"873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57\": container with ID starting with 873a0906c9965ab7f0dc757652b5cdd06c66e324cf3e33617f27a025b400dc57 not found: ID does not exist"
Nov 02 12:57:45.584986 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.584908    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"]
Nov 02 12:57:45.590758 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.590727    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-74ba71-576f84f849-68zpj"]
Nov 02 12:57:45.604459 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:45.604431    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbp86\" (UniqueName: \"kubernetes.io/projected/99f9f86a-b60e-4b2b-8535-c28fd981e6e6-kube-api-access-wbp86\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:57:46.018024 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:46.017976    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f9f86a-b60e-4b2b-8535-c28fd981e6e6" path="/var/lib/kubelet/pods/99f9f86a-b60e-4b2b-8535-c28fd981e6e6/volumes"
Nov 02 12:57:46.263026 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:46.262960    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:46.619338 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:46.619305    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:47.033715 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:47.033632    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"]
Nov 02 12:57:47.046958 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:47.046932    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada1da4c-zc6tv"]
Nov 02 12:57:48.018132 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:48.018102    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2b2b22e-5109-46e7-9cfe-2ef93976715d" path="/var/lib/kubelet/pods/f2b2b22e-5109-46e7-9cfe-2ef93976715d/volumes"
Nov 02 12:57:48.135951 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:48.135917    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:49.736454 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:49.736422    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:51.257527 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:51.257492    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:52.877890 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:52.877849    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:54.034308 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:54.034280    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"]
Nov 02 12:57:54.059173 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:54.059142    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1193ef8f-wh42v"]
Nov 02 12:57:54.475235 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:54.475204    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:55.987400 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:55.987363    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:56.028449 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:56.028410    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eab034c-f96c-4890-a455-164c4e27bad7" path="/var/lib/kubelet/pods/2eab034c-f96c-4890-a455-164c4e27bad7/volumes"
Nov 02 12:57:57.569304 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:57.569261    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:57:59.155023 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:57:59.154961    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:00.738518 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:00.738467    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:02.035168 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:02.035125    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"]
Nov 02 12:58:02.048744 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:02.048709    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54b69fec-nqlxh"]
Nov 02 12:58:02.056934 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:02.056900    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:02.913586 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:02.913536    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:04.028843 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:04.028799    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a60794c-3304-4ea0-9a34-06e86b1be8eb" path="/var/lib/kubelet/pods/2a60794c-3304-4ea0-9a34-06e86b1be8eb/volumes"
Nov 02 12:58:04.455484 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:04.455453    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:05.033549 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:05.033518    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"]
Nov 02 12:58:05.046907 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:05.046877    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d4cfbb3-6pgtg"]
Nov 02 12:58:05.956356 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:05.956316    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:06.022137 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:06.022097    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf104234-fe37-408f-9abd-9454197750a8" path="/var/lib/kubelet/pods/cf104234-fe37-408f-9abd-9454197750a8/volumes"
Nov 02 12:58:07.547842 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:07.547810    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:09.195911 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:09.195865    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:10.524308 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:10.524275    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:10.654300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:10.654266    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:10.770350 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:10.770316    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:11.266146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:11.266110    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:11.675123 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:11.675088    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:12.667670 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:12.667630    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:13.032919 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:13.032837    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"]
Nov 02 12:58:13.046547 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:13.046515    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be3f37f0-k6tt7"]
Nov 02 12:58:13.059675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:13.059643    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:13.536258 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:13.536226    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:13.719490 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:13.719457    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:14.018913 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:14.018881    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="959aa0ec-aca6-4538-85b9-f7ee508469fe" path="/var/lib/kubelet/pods/959aa0ec-aca6-4538-85b9-f7ee508469fe/volumes"
Nov 02 12:58:14.444746 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:14.444714    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:14.821546 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:58:14.821457    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:58:16.034318 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:16.034279    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"]
Nov 02 12:58:16.048929 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:16.048895    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9b21fdf-7mln6"]
Nov 02 12:58:16.072385 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:16.072351    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:17.577027 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:17.576973    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:17.705119 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:17.705058    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:17.945223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:17.945190    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:18.024569 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:18.024530    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b79524e-94e3-4b98-8835-f7f7b59c7b1c" path="/var/lib/kubelet/pods/1b79524e-94e3-4b98-8835-f7f7b59c7b1c/volumes"
Nov 02 12:58:19.030608 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:19.030578    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"]
Nov 02 12:58:19.044272 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:19.044247    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89eb17ec-dcb2c"]
Nov 02 12:58:19.253600 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:19.253560    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:19.385693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:19.385649    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:19.952957 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:19.952924    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:20.019614 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:20.019552    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60a77211-86ad-4403-80f5-37863af1252f" path="/var/lib/kubelet/pods/60a77211-86ad-4403-80f5-37863af1252f/volumes"
Nov 02 12:58:21.557221 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:21.557180    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:23.109567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:23.109531    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:24.630557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:24.630523    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:26.145095 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:26.145063    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:27.041312 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:27.041272    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"]
Nov 02 12:58:27.044759 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:27.044731    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73b9582-8zvcj"]
Nov 02 12:58:27.731457 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:27.731420    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:28.018471 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:28.018386    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809fd484-fc6d-4e39-9fe0-2f7d071bd6ba" path="/var/lib/kubelet/pods/809fd484-fc6d-4e39-9fe0-2f7d071bd6ba/volumes"
Nov 02 12:58:29.348857 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:29.348825    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:30.940673 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:30.940638    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:32.508709 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:32.508659    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:34.053207 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:34.053171    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:35.308127 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:35.308091    5879 scope.go:122] "RemoveContainer" containerID="e025a956cb4cc9fa463557d8f24ee173ea81d195d5a69036ac0c446e46388edb"
Nov 02 12:58:35.319503 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:35.319478    5879 scope.go:122] "RemoveContainer" containerID="5b249fc9d287b0d07dc964c9b462cc3e06cd3e30a1368470ef8a23d8001ff6ad"
Nov 02 12:58:35.329186 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:35.329162    5879 scope.go:122] "RemoveContainer" containerID="e4560a662aae08e0101ae15692ab2c7a10731639deaf866b6ce3dcaadec338c2"
Nov 02 12:58:35.339267 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:35.339240    5879 scope.go:122] "RemoveContainer" containerID="82a5e412a600048f1921c1a045898ccb0f1700871e8da570da2f63b1014aa0b6"
Nov 02 12:58:35.348662 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:35.348642    5879 scope.go:122] "RemoveContainer" containerID="b422bf728d761e2d9912ca166b7e327debaf716d84c1bc441bfe296f0fb59280"
Nov 02 12:58:35.357878 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:35.357854    5879 scope.go:122] "RemoveContainer" containerID="19138d1e0cfcf0ded571850e3e64c5cfdaac762d8ac8f4144ecc6f3174450d32"
Nov 02 12:58:35.366908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:35.366891    5879 scope.go:122] "RemoveContainer" containerID="a2b2480333e2fd070f2fdaf9ebdcf7be6777c762620373b5de92c161fcdd0f25"
Nov 02 12:58:35.376663 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:35.376645    5879 scope.go:122] "RemoveContainer" containerID="48ce97e16de4723bf7d53dfcb15ab806e4ca0d5cff8dc9106cd587ce8c61b874"
Nov 02 12:58:35.387932 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:35.387910    5879 scope.go:122] "RemoveContainer" containerID="3e345b9f21ff5ccab9ea2813a7d1f161194067897cedf06e9f01444c75108028"
Nov 02 12:58:35.396874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:35.396848    5879 scope.go:122] "RemoveContainer" containerID="b61840a347f356b073db776648dd954f03843930de6cf3ba54adf4b0ac1f7455"
Nov 02 12:58:35.527653 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:35.527618    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:36.940557 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:36.940523    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:37.112980 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:37.112944    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:38.654980 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:38.654949    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:40.217733 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:40.217701    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:40.357667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:40.349017    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:40.472854 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:40.472771    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:41.926839 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:41.926749    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:43.032193 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:43.032158    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"]
Nov 02 12:58:43.045471 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:43.045441    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eea18e0-92vxx"]
Nov 02 12:58:43.624446 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:43.624414    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:44.017805 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:44.017721    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e73b6d7-d744-4f73-b1b5-dd3437ad6888" path="/var/lib/kubelet/pods/6e73b6d7-d744-4f73-b1b5-dd3437ad6888/volumes"
Nov 02 12:58:44.820538 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:58:44.820499    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:58:45.139195 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:45.139164    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:46.788193 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:46.788160    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:48.516253 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:48.516216    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:50.219398 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:50.219367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:51.979222 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:51.979193    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:53.090947 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:53.090909    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"]
Nov 02 12:58:53.091417 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:53.091129    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy" containerID="cri-o://f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc" gracePeriod=30
Nov 02 12:58:53.548073 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:53.548039    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:55.140073 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:55.140023    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:56.713060 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:56.713028    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:58.227034 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.226970    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:58:58.391768 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.391742    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:58:58.518454 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.518420    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-socket\") pod \"3b691630-bc7c-4785-939f-7eeb342f5264\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") "
Nov 02 12:58:58.518668 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.518498    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-istio-token\" (UniqueName: \"kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-istio-token\") pod \"3b691630-bc7c-4785-939f-7eeb342f5264\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") "
Nov 02 12:58:58.518668 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.518608    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-credential-socket\") pod \"3b691630-bc7c-4785-939f-7eeb342f5264\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") "
Nov 02 12:58:58.518802 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.518680    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-envoy\") pod \"3b691630-bc7c-4785-939f-7eeb342f5264\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") "
Nov 02 12:58:58.518802 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.518741    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-certs\") pod \"3b691630-bc7c-4785-939f-7eeb342f5264\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") "
Nov 02 12:58:58.518914 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.518843    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b691630-bc7c-4785-939f-7eeb342f5264-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b691630-bc7c-4785-939f-7eeb342f5264-istio-podinfo\") pod \"3b691630-bc7c-4785-939f-7eeb342f5264\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") "
Nov 02 12:58:58.518914 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.518896    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-kube-api-access-hkn6t\" (UniqueName: \"kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-kube-api-access-hkn6t\") pod \"3b691630-bc7c-4785-939f-7eeb342f5264\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") "
Nov 02 12:58:58.519070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.518940    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b691630-bc7c-4785-939f-7eeb342f5264-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b691630-bc7c-4785-939f-7eeb342f5264-istiod-ca-cert\") pod \"3b691630-bc7c-4785-939f-7eeb342f5264\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") "
Nov 02 12:58:58.519070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.518981    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-data\") pod \"3b691630-bc7c-4785-939f-7eeb342f5264\" (UID: \"3b691630-bc7c-4785-939f-7eeb342f5264\") "
Nov 02 12:58:58.519456 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.519408    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-certs" pod "3b691630-bc7c-4785-939f-7eeb342f5264" (UID: "3b691630-bc7c-4785-939f-7eeb342f5264"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:58.519607 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.519490    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-socket" pod "3b691630-bc7c-4785-939f-7eeb342f5264" (UID: "3b691630-bc7c-4785-939f-7eeb342f5264"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:58.519763 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.519738    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-data" pod "3b691630-bc7c-4785-939f-7eeb342f5264" (UID: "3b691630-bc7c-4785-939f-7eeb342f5264"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:58.519834 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.519791    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-credential-socket" pod "3b691630-bc7c-4785-939f-7eeb342f5264" (UID: "3b691630-bc7c-4785-939f-7eeb342f5264"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:58.519907 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.519826    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b691630-bc7c-4785-939f-7eeb342f5264-istiod-ca-cert" pod "3b691630-bc7c-4785-939f-7eeb342f5264" (UID: "3b691630-bc7c-4785-939f-7eeb342f5264"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:58:58.523773 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.523747    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b691630-bc7c-4785-939f-7eeb342f5264-istio-podinfo" pod "3b691630-bc7c-4785-939f-7eeb342f5264" (UID: "3b691630-bc7c-4785-939f-7eeb342f5264"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:58:58.525329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.525298    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-envoy" pod "3b691630-bc7c-4785-939f-7eeb342f5264" (UID: "3b691630-bc7c-4785-939f-7eeb342f5264"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:58:58.525439 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.525344    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-istio-token" pod "3b691630-bc7c-4785-939f-7eeb342f5264" (UID: "3b691630-bc7c-4785-939f-7eeb342f5264"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:58.526913 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.526882    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-kube-api-access-hkn6t" pod "3b691630-bc7c-4785-939f-7eeb342f5264" (UID: "3b691630-bc7c-4785-939f-7eeb342f5264"). InnerVolumeSpecName "kube-api-access-hkn6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:58:58.621130 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.621098    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.621130 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.621131    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.621394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.621143    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.621394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.621155    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.621394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.621166    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b691630-bc7c-4785-939f-7eeb342f5264-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.621394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.621179    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkn6t\" (UniqueName: \"kubernetes.io/projected/3b691630-bc7c-4785-939f-7eeb342f5264-kube-api-access-hkn6t\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.621394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.621192    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b691630-bc7c-4785-939f-7eeb342f5264-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.621394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.621202    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:58:58.621394 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:58.621214    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b691630-bc7c-4785-939f-7eeb342f5264-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:58:59.139691 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:59.139654    5879 generic.go:356] "Generic (PLEG): container finished" podID="3b691630-bc7c-4785-939f-7eeb342f5264" containerID="f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc" exitCode=0
Nov 02 12:58:59.139691 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:59.139701    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"
Nov 02 12:58:59.139939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:59.139736    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" event={"ID":"3b691630-bc7c-4785-939f-7eeb342f5264","Type":"ContainerDied","Data":"f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc"}
Nov 02 12:58:59.139939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:59.139786    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9" event={"ID":"3b691630-bc7c-4785-939f-7eeb342f5264","Type":"ContainerDied","Data":"2a61f51142ee66344348c8a9a385e5ee8749ba412020d463a77b9f540720c93d"}
Nov 02 12:58:59.139939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:59.139808    5879 scope.go:122] "RemoveContainer" containerID="f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc"
Nov 02 12:58:59.151588 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:59.151572    5879 scope.go:122] "RemoveContainer" containerID="f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc"
Nov 02 12:58:59.151898 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:58:59.151878    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc\": container with ID starting with f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc not found: ID does not exist" containerID="f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc"
Nov 02 12:58:59.151953 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:59.151907    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc"} err="failed to get container status \"f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc\": rpc error: code = NotFound desc = could not find container \"f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc\": container with ID starting with f59cb50bd3c6f8ee32fbadfe9e6640ccfb7aa8edeeb2ed809d5b48bb4cb779dc not found: ID does not exist"
Nov 02 12:58:59.211765 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:59.211735    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"]
Nov 02 12:58:59.216584 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:59.216532    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-bbgw9"]
Nov 02 12:58:59.773241 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:58:59.773202    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:00.017682 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:00.017647    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" path="/var/lib/kubelet/pods/3b691630-bc7c-4785-939f-7eeb342f5264/volumes"
Nov 02 12:59:01.355414 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:01.355369    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:02.954532 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:02.954502    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:04.637426 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:04.637393    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:06.210797 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:06.210761    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:07.836674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:07.836633    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:08.500796 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:08.500755    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:09.637235 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:09.637198    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:10.092125 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:10.092024    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:11.246637 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:11.246604    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:12.854601 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:12.854567    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:14.533543 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:14.533459    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:14.822881 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:59:14.822790    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:59:16.111693 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:16.111652    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:17.642269 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:17.642229    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:19.186883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:19.186849    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:20.911276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:20.911238    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:21.927223 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:21.927191    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:22.040397 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:22.040367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:22.638718 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:22.638686    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:24.148935 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:24.148905    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:25.718567 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:25.718528    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:27.298317 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:27.298285    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:28.768276 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.768233    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"]
Nov 02 12:59:28.769112 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.769090    5879 cpu_manager.go:414] "containerMap: removing container" podUID="99f9f86a-b60e-4b2b-8535-c28fd981e6e6" containerName="payload-processing"
Nov 02 12:59:28.769112 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.769111    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f9f86a-b60e-4b2b-8535-c28fd981e6e6" containerName="payload-processing"
Nov 02 12:59:28.769300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.769143    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9d4fdad1-9361-4f4e-8f4d-d68dae46d602" containerName="revoke-keys"
Nov 02 12:59:28.769300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.769150    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d4fdad1-9361-4f4e-8f4d-d68dae46d602" containerName="revoke-keys"
Nov 02 12:59:28.769300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.769167    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy"
Nov 02 12:59:28.769300 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.769175    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy"
Nov 02 12:59:28.769508 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.769379    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f9f86a-b60e-4b2b-8535-c28fd981e6e6" containerName="payload-processing"
Nov 02 12:59:28.769508 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.769396    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d4fdad1-9361-4f4e-8f4d-d68dae46d602" containerName="revoke-keys"
Nov 02 12:59:28.769508 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.769407    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b691630-bc7c-4785-939f-7eeb342f5264" containerName="istio-proxy"
Nov 02 12:59:28.773178 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.773155    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:28.793070 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.793038    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"]
Nov 02 12:59:28.802104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.802063    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:59:28.802271 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.802201    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:59:28.879787 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.879745    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50f7f72e-xk5lp\" (UID: \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:28.880017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.879948    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxfk5\" (UniqueName: \"kubernetes.io/projected/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-kube-api-access-rxfk5\") pod \"maas-api-revoke-sub-50f7f72e-xk5lp\" (UID: \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:28.880105 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.880088    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50f7f72e-xk5lp\" (UID: \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:28.885661 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.885635    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:28.981319 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.981283    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50f7f72e-xk5lp\" (UID: \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:28.981535 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.981452    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50f7f72e-xk5lp\" (UID: \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:28.981634 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.981605    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxfk5\" (UniqueName: \"kubernetes.io/projected/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-kube-api-access-rxfk5\") pod \"maas-api-revoke-sub-50f7f72e-xk5lp\" (UID: \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:28.982391 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.982365    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50f7f72e-xk5lp\" (UID: \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:28.983880 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:28.983848    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50f7f72e-xk5lp\" (UID: \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:29.007297 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:29.007261    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxfk5\" (UniqueName: \"kubernetes.io/projected/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-kube-api-access-rxfk5\") pod \"maas-api-revoke-sub-50f7f72e-xk5lp\" (UID: \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:29.087200 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:29.087118    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:29.395332 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:29.395283    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"]
Nov 02 12:59:30.405978 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:30.405948    5879 generic.go:356] "Generic (PLEG): container finished" podID="05a0193f-86ca-4a9c-aa04-fd0d80a1743b" containerID="ab1f7d63a0dd4eddcd9ef13b55c72829fde53c9d2090429619c968705c58eeb4" exitCode=0
Nov 02 12:59:30.406365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:30.406052    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp" event={"ID":"05a0193f-86ca-4a9c-aa04-fd0d80a1743b","Type":"ContainerDied","Data":"ab1f7d63a0dd4eddcd9ef13b55c72829fde53c9d2090429619c968705c58eeb4"}
Nov 02 12:59:30.406365 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:30.406112    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp" event={"ID":"05a0193f-86ca-4a9c-aa04-fd0d80a1743b","Type":"ContainerStarted","Data":"cbfdfb7ae9ad1bdf8f3d6605a4ac983ee9855842beeec2da366341cb29ff8634"}
Nov 02 12:59:30.445713 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:30.445664    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:31.032979 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.032938    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"]
Nov 02 12:59:31.037705 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.037679    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:31.057583 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.057553    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"]
Nov 02 12:59:31.115860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.115821    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85ef3ba1-z65j9\" (UID: \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:31.116126 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.115884    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gwcg\" (UniqueName: \"kubernetes.io/projected/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-kube-api-access-7gwcg\") pod \"maas-api-revoke-sub-85ef3ba1-z65j9\" (UID: \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:31.116212 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.116148    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85ef3ba1-z65j9\" (UID: \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:31.217757 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.217727    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85ef3ba1-z65j9\" (UID: \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:31.217944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.217770    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gwcg\" (UniqueName: \"kubernetes.io/projected/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-kube-api-access-7gwcg\") pod \"maas-api-revoke-sub-85ef3ba1-z65j9\" (UID: \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:31.217944 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.217830    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85ef3ba1-z65j9\" (UID: \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:31.218434 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.218411    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85ef3ba1-z65j9\" (UID: \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:31.220264 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.220240    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85ef3ba1-z65j9\" (UID: \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:31.242688 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.242649    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gwcg\" (UniqueName: \"kubernetes.io/projected/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-kube-api-access-7gwcg\") pod \"maas-api-revoke-sub-85ef3ba1-z65j9\" (UID: \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:31.357182 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.357092    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:31.422862 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.422793    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"]
Nov 02 12:59:31.423287 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.423057    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" podUID="66efff16-b760-4e92-89de-c51ed4a48087" containerName="istio-proxy" containerID="cri-o://663c3990db661a7dcd730c6591a857b7747bcf931ea3069288ee4f99fe718a9d" gracePeriod=30
Nov 02 12:59:31.625669 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.625642    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:31.673081 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:59:31.673040    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572bcf39_cd03_4a12_a5dc_a2e8dca2d872.slice/crio-91982c270d6a06ff6966e51a467575727804aa14c64f00a3ae8de6fbe71cd69b WatchSource:0}: Error finding container 91982c270d6a06ff6966e51a467575727804aa14c64f00a3ae8de6fbe71cd69b: Status 404 returned error can't find the container with id 91982c270d6a06ff6966e51a467575727804aa14c64f00a3ae8de6fbe71cd69b
Nov 02 12:59:31.673722 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.673690    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"]
Nov 02 12:59:31.725398 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.725368    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-openshift-ca\") pod \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\" (UID: \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\") "
Nov 02 12:59:31.725588 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.725570    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-xks-ca\") pod \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\" (UID: \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\") "
Nov 02 12:59:31.725667 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.725599    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-kube-api-access-rxfk5\" (UniqueName: \"kubernetes.io/projected/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-kube-api-access-rxfk5\") pod \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\" (UID: \"05a0193f-86ca-4a9c-aa04-fd0d80a1743b\") "
Nov 02 12:59:31.725809 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.725787    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-openshift-ca" pod "05a0193f-86ca-4a9c-aa04-fd0d80a1743b" (UID: "05a0193f-86ca-4a9c-aa04-fd0d80a1743b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:31.726169 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.726149    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:31.727860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.727838    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-xks-ca" pod "05a0193f-86ca-4a9c-aa04-fd0d80a1743b" (UID: "05a0193f-86ca-4a9c-aa04-fd0d80a1743b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:59:31.727941 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.727920    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-kube-api-access-rxfk5" pod "05a0193f-86ca-4a9c-aa04-fd0d80a1743b" (UID: "05a0193f-86ca-4a9c-aa04-fd0d80a1743b"). InnerVolumeSpecName "kube-api-access-rxfk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:31.827836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.827796    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:31.827836 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:31.827841    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxfk5\" (UniqueName: \"kubernetes.io/projected/05a0193f-86ca-4a9c-aa04-fd0d80a1743b-kube-api-access-rxfk5\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:32.119173 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:32.119142    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:32.422969 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:32.422926    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9" event={"ID":"572bcf39-cd03-4a12-a5dc-a2e8dca2d872","Type":"ContainerStarted","Data":"e1325d3484a90445e72f6ad09220cf16d25729e9f3e56a3e7c853a2d08db1a37"}
Nov 02 12:59:32.423495 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:32.423010    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9" event={"ID":"572bcf39-cd03-4a12-a5dc-a2e8dca2d872","Type":"ContainerStarted","Data":"91982c270d6a06ff6966e51a467575727804aa14c64f00a3ae8de6fbe71cd69b"}
Nov 02 12:59:32.424600 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:32.424573    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"
Nov 02 12:59:32.424600 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:32.424584    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp" event={"ID":"05a0193f-86ca-4a9c-aa04-fd0d80a1743b","Type":"ContainerDied","Data":"cbfdfb7ae9ad1bdf8f3d6605a4ac983ee9855842beeec2da366341cb29ff8634"}
Nov 02 12:59:32.424767 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:32.424623    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbfdfb7ae9ad1bdf8f3d6605a4ac983ee9855842beeec2da366341cb29ff8634"
Nov 02 12:59:32.468376 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:32.468326    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9" podStartSLOduration=1.468312247 podStartE2EDuration="1.468312247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:59:32.466413391 +0000 UTC m=+1918.573116283" watchObservedRunningTime="2026-11-02 12:59:32.468312247 +0000 UTC m=+1918.575015138"
Nov 02 12:59:33.434103 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:33.434070    5879 generic.go:356] "Generic (PLEG): container finished" podID="572bcf39-cd03-4a12-a5dc-a2e8dca2d872" containerID="e1325d3484a90445e72f6ad09220cf16d25729e9f3e56a3e7c853a2d08db1a37" exitCode=0
Nov 02 12:59:33.434520 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:33.434150    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9" event={"ID":"572bcf39-cd03-4a12-a5dc-a2e8dca2d872","Type":"ContainerDied","Data":"e1325d3484a90445e72f6ad09220cf16d25729e9f3e56a3e7c853a2d08db1a37"}
Nov 02 12:59:33.750566 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:33.750482    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:34.323326 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:34.323285    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"]
Nov 02 12:59:34.323563 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:34.323521    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerName="istio-proxy" containerID="cri-o://673e0159cf29ef458c209aedd01897298a6cfa0f0986d64c7ee257f83577016c" gracePeriod=30
Nov 02 12:59:34.612704 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:34.612675    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:34.681723 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:34.681690    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-xks-ca\") pod \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\" (UID: \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\") "
Nov 02 12:59:34.681922 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:34.681747    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-kube-api-access-7gwcg\" (UniqueName: \"kubernetes.io/projected/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-kube-api-access-7gwcg\") pod \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\" (UID: \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\") "
Nov 02 12:59:34.684476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:34.684443    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-xks-ca" pod "572bcf39-cd03-4a12-a5dc-a2e8dca2d872" (UID: "572bcf39-cd03-4a12-a5dc-a2e8dca2d872"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:59:34.684661 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:34.684638    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-kube-api-access-7gwcg" pod "572bcf39-cd03-4a12-a5dc-a2e8dca2d872" (UID: "572bcf39-cd03-4a12-a5dc-a2e8dca2d872"). InnerVolumeSpecName "kube-api-access-7gwcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:34.783415 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:34.783382    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-openshift-ca\") pod \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\" (UID: \"572bcf39-cd03-4a12-a5dc-a2e8dca2d872\") "
Nov 02 12:59:34.783699 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:34.783678    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-openshift-ca" pod "572bcf39-cd03-4a12-a5dc-a2e8dca2d872" (UID: "572bcf39-cd03-4a12-a5dc-a2e8dca2d872"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:34.783902 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:34.783885    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:34.783939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:34.783908    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:34.783939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:34.783917    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gwcg\" (UniqueName: \"kubernetes.io/projected/572bcf39-cd03-4a12-a5dc-a2e8dca2d872-kube-api-access-7gwcg\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:35.187652 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.187604    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"]
Nov 02 12:59:35.188082 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.187959    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6" podUID="280bc232-ca40-4246-ac92-5cda8dac9930" containerName="discovery" containerID="cri-o://6767b8fe8de377acd36d989bade1de492d7a7dc3b5e6aae6fda9578d266219e3" gracePeriod=30
Nov 02 12:59:35.208044 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.205429    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:35.272153 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:59:35.272104    5879 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280bc232_ca40_4246_ac92_5cda8dac9930.slice/crio-conmon-6767b8fe8de377acd36d989bade1de492d7a7dc3b5e6aae6fda9578d266219e3.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 12:59:35.359408 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.359374    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:35.482865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.474329    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"
Nov 02 12:59:35.504951 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.468104    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9" event={"ID":"572bcf39-cd03-4a12-a5dc-a2e8dca2d872","Type":"ContainerDied","Data":"91982c270d6a06ff6966e51a467575727804aa14c64f00a3ae8de6fbe71cd69b"}
Nov 02 12:59:35.504951 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.504796    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91982c270d6a06ff6966e51a467575727804aa14c64f00a3ae8de6fbe71cd69b"
Nov 02 12:59:35.514650 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.514624    5879 scope.go:122] "RemoveContainer" containerID="d5ced8d981c8d5f68261ff7ac78ef96afb28de02f64b633276b59a933e77c47b"
Nov 02 12:59:35.559189 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.559151    5879 generic.go:356] "Generic (PLEG): container finished" podID="280bc232-ca40-4246-ac92-5cda8dac9930" containerID="6767b8fe8de377acd36d989bade1de492d7a7dc3b5e6aae6fda9578d266219e3" exitCode=0
Nov 02 12:59:35.559390 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.559267    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6" event={"ID":"280bc232-ca40-4246-ac92-5cda8dac9930","Type":"ContainerDied","Data":"6767b8fe8de377acd36d989bade1de492d7a7dc3b5e6aae6fda9578d266219e3"}
Nov 02 12:59:35.571638 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.571611    5879 scope.go:122] "RemoveContainer" containerID="0f543e5bd60c1fef3e7979c0ab6854e0d2412615216aa697e09c91dd638c3964"
Nov 02 12:59:35.577703 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.577674    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:59:35.584399 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.584371    5879 scope.go:122] "RemoveContainer" containerID="e5fa1a17e0168c29a943ffc0b1e5b083c9253af160ad6e3768131dbd55c9d45b"
Nov 02 12:59:35.599497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.598741    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-ca-configmap\") pod \"280bc232-ca40-4246-ac92-5cda8dac9930\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") "
Nov 02 12:59:35.599497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.598871    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-dns-cert\") pod \"280bc232-ca40-4246-ac92-5cda8dac9930\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") "
Nov 02 12:59:35.599497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.598909    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-cacerts\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-cacerts\") pod \"280bc232-ca40-4246-ac92-5cda8dac9930\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") "
Nov 02 12:59:35.599497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.598957    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-kubeconfig\") pod \"280bc232-ca40-4246-ac92-5cda8dac9930\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") "
Nov 02 12:59:35.599497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.599017    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-istio-token\" (UniqueName: \"kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-istio-token\") pod \"280bc232-ca40-4246-ac92-5cda8dac9930\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") "
Nov 02 12:59:35.599497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.599071    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280bc232-ca40-4246-ac92-5cda8dac9930-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/280bc232-ca40-4246-ac92-5cda8dac9930-local-certs\") pod \"280bc232-ca40-4246-ac92-5cda8dac9930\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") "
Nov 02 12:59:35.599497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.599212    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-ca-configmap" pod "280bc232-ca40-4246-ac92-5cda8dac9930" (UID: "280bc232-ca40-4246-ac92-5cda8dac9930"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:35.599497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.599247    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-kube-api-access-qpztn\" (UniqueName: \"kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-kube-api-access-qpztn\") pod \"280bc232-ca40-4246-ac92-5cda8dac9930\" (UID: \"280bc232-ca40-4246-ac92-5cda8dac9930\") "
Nov 02 12:59:35.600484 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.599870    5879 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-ca-configmap\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:35.601982 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.601956    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-istio-token" pod "280bc232-ca40-4246-ac92-5cda8dac9930" (UID: "280bc232-ca40-4246-ac92-5cda8dac9930"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:35.605150 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.605122    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280bc232-ca40-4246-ac92-5cda8dac9930-local-certs" pod "280bc232-ca40-4246-ac92-5cda8dac9930" (UID: "280bc232-ca40-4246-ac92-5cda8dac9930"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:35.605150 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.605140    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-kube-api-access-qpztn" pod "280bc232-ca40-4246-ac92-5cda8dac9930" (UID: "280bc232-ca40-4246-ac92-5cda8dac9930"). InnerVolumeSpecName "kube-api-access-qpztn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:35.605303 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.605146    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-cacerts" pod "280bc232-ca40-4246-ac92-5cda8dac9930" (UID: "280bc232-ca40-4246-ac92-5cda8dac9930"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:59:35.605303 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.605121    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-kubeconfig" pod "280bc232-ca40-4246-ac92-5cda8dac9930" (UID: "280bc232-ca40-4246-ac92-5cda8dac9930"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:59:35.605303 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.605283    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-dns-cert" pod "280bc232-ca40-4246-ac92-5cda8dac9930" (UID: "280bc232-ca40-4246-ac92-5cda8dac9930"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:59:35.606712 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.606690    5879 scope.go:122] "RemoveContainer" containerID="1aab9d948597b872ceea3aac15956a80188aef9afa70313cd029de67eb77aa9d"
Nov 02 12:59:35.636697 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.636676    5879 scope.go:122] "RemoveContainer" containerID="c988e1dcbe85312560ffcf135423cf26c46e6badff3aeae987cdaed85ebbc0b9"
Nov 02 12:59:35.650820 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.650787    5879 scope.go:122] "RemoveContainer" containerID="6767b8fe8de377acd36d989bade1de492d7a7dc3b5e6aae6fda9578d266219e3"
Nov 02 12:59:35.662604 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.662567    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:35.667970 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.667949    5879 scope.go:122] "RemoveContainer" containerID="63f06613677a7d2efdcd9819bf7bffb68b6408b647660e95e32013ed95bf664a"
Nov 02 12:59:35.678446 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.678420    5879 scope.go:122] "RemoveContainer" containerID="10075cd7ba35d13fabe36d48bc122c2081486fea7aa326c17674d1e037d68468"
Nov 02 12:59:35.688588 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.688566    5879 scope.go:122] "RemoveContainer" containerID="e87bfc22abc04a06069a45d6dc8f419e5be5662ae515df091aeb088a8936d634"
Nov 02 12:59:35.698478 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.698455    5879 scope.go:122] "RemoveContainer" containerID="66c6f281dbb2dc0a83d0f56b49df28034f2adafe770fc471fe42843e005bc7f5"
Nov 02 12:59:35.701806 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.701782    5879 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-csr-dns-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:35.701898 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.701810    5879 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-cacerts\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:35.701898 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.701829    5879 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/280bc232-ca40-4246-ac92-5cda8dac9930-istio-kubeconfig\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:35.701898 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.701844    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:35.701898 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.701858    5879 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/280bc232-ca40-4246-ac92-5cda8dac9930-local-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:35.701898 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:35.701872    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpztn\" (UniqueName: \"kubernetes.io/projected/280bc232-ca40-4246-ac92-5cda8dac9930-kube-api-access-qpztn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.565809 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.565774    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6" event={"ID":"280bc232-ca40-4246-ac92-5cda8dac9930","Type":"ContainerDied","Data":"4bc10f01314793438cb1fcf83e8bd4939ba5c0f15da5e5653e5268a2e2c5eb35"}
Nov 02 12:59:36.570746 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.570685    5879 generic.go:356] "Generic (PLEG): container finished" podID="66efff16-b760-4e92-89de-c51ed4a48087" containerID="663c3990db661a7dcd730c6591a857b7747bcf931ea3069288ee4f99fe718a9d" exitCode=0
Nov 02 12:59:36.570968 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.570825    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"
Nov 02 12:59:36.580453 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.570982    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" event={"ID":"66efff16-b760-4e92-89de-c51ed4a48087","Type":"ContainerDied","Data":"663c3990db661a7dcd730c6591a857b7747bcf931ea3069288ee4f99fe718a9d"}
Nov 02 12:59:36.683104 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.683065    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"]
Nov 02 12:59:36.684664 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.684638    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vzlw6"]
Nov 02 12:59:36.748448 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.748422    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:59:36.817939 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.817890    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-envoy\") pod \"66efff16-b760-4e92-89de-c51ed4a48087\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") "
Nov 02 12:59:36.818151 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.817956    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-socket\") pod \"66efff16-b760-4e92-89de-c51ed4a48087\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") "
Nov 02 12:59:36.818151 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.818005    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66efff16-b760-4e92-89de-c51ed4a48087-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66efff16-b760-4e92-89de-c51ed4a48087-istiod-ca-cert\") pod \"66efff16-b760-4e92-89de-c51ed4a48087\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") "
Nov 02 12:59:36.818306 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.818165    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-credential-socket\") pod \"66efff16-b760-4e92-89de-c51ed4a48087\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") "
Nov 02 12:59:36.818306 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.818287    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-kube-api-access-k77pm\" (UniqueName: \"kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-kube-api-access-k77pm\") pod \"66efff16-b760-4e92-89de-c51ed4a48087\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") "
Nov 02 12:59:36.818417 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.818324    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-certs\") pod \"66efff16-b760-4e92-89de-c51ed4a48087\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") "
Nov 02 12:59:36.818417 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.818396    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66efff16-b760-4e92-89de-c51ed4a48087-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66efff16-b760-4e92-89de-c51ed4a48087-istio-podinfo\") pod \"66efff16-b760-4e92-89de-c51ed4a48087\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") "
Nov 02 12:59:36.818523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.818433    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-data\") pod \"66efff16-b760-4e92-89de-c51ed4a48087\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") "
Nov 02 12:59:36.818523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.818442    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-credential-socket" pod "66efff16-b760-4e92-89de-c51ed4a48087" (UID: "66efff16-b760-4e92-89de-c51ed4a48087"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.818523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.818508    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-istio-token\" (UniqueName: \"kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-istio-token\") pod \"66efff16-b760-4e92-89de-c51ed4a48087\" (UID: \"66efff16-b760-4e92-89de-c51ed4a48087\") "
Nov 02 12:59:36.819399 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.818816    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-data" pod "66efff16-b760-4e92-89de-c51ed4a48087" (UID: "66efff16-b760-4e92-89de-c51ed4a48087"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.819399 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.818864    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66efff16-b760-4e92-89de-c51ed4a48087-istiod-ca-cert" pod "66efff16-b760-4e92-89de-c51ed4a48087" (UID: "66efff16-b760-4e92-89de-c51ed4a48087"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:36.819399 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.818929    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-certs" pod "66efff16-b760-4e92-89de-c51ed4a48087" (UID: "66efff16-b760-4e92-89de-c51ed4a48087"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.819399 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.819368    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-socket" pod "66efff16-b760-4e92-89de-c51ed4a48087" (UID: "66efff16-b760-4e92-89de-c51ed4a48087"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.819822 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.819553    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.819822 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.819572    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.819822 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.819587    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.819822 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.819599    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.819822 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.819612    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66efff16-b760-4e92-89de-c51ed4a48087-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.821860 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.821836    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-envoy" pod "66efff16-b760-4e92-89de-c51ed4a48087" (UID: "66efff16-b760-4e92-89de-c51ed4a48087"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:36.821967 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.821927    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66efff16-b760-4e92-89de-c51ed4a48087-istio-podinfo" pod "66efff16-b760-4e92-89de-c51ed4a48087" (UID: "66efff16-b760-4e92-89de-c51ed4a48087"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:36.821967 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.821955    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-istio-token" pod "66efff16-b760-4e92-89de-c51ed4a48087" (UID: "66efff16-b760-4e92-89de-c51ed4a48087"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:36.823680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.823659    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-kube-api-access-k77pm" pod "66efff16-b760-4e92-89de-c51ed4a48087" (UID: "66efff16-b760-4e92-89de-c51ed4a48087"). InnerVolumeSpecName "kube-api-access-k77pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:36.920819 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.920784    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66efff16-b760-4e92-89de-c51ed4a48087-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.920819 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.920815    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.920819 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.920825    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66efff16-b760-4e92-89de-c51ed4a48087-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:36.921207 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:36.920835    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k77pm\" (UniqueName: \"kubernetes.io/projected/66efff16-b760-4e92-89de-c51ed4a48087-kube-api-access-k77pm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:37.052337 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:37.052252    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:37.176431 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:37.176374    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:37.300971 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:37.300931    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:37.579830 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:37.579802    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"
Nov 02 12:59:37.579830 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:37.579825    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g" event={"ID":"66efff16-b760-4e92-89de-c51ed4a48087","Type":"ContainerDied","Data":"13890ccd4681c1b0519ce55e4399de476daadba8ebd40ee3c34ee13e5acb3b25"}
Nov 02 12:59:37.580068 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:37.579895    5879 scope.go:122] "RemoveContainer" containerID="663c3990db661a7dcd730c6591a857b7747bcf931ea3069288ee4f99fe718a9d"
Nov 02 12:59:37.636112 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:37.636082    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"]
Nov 02 12:59:37.649950 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:37.649919    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-2lf6g"]
Nov 02 12:59:38.018261 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:38.018225    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280bc232-ca40-4246-ac92-5cda8dac9930" path="/var/lib/kubelet/pods/280bc232-ca40-4246-ac92-5cda8dac9930/volumes"
Nov 02 12:59:38.018816 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:38.018797    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66efff16-b760-4e92-89de-c51ed4a48087" path="/var/lib/kubelet/pods/66efff16-b760-4e92-89de-c51ed4a48087/volumes"
Nov 02 12:59:38.630265 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:38.630222    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:39.157704 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.157672    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"]
Nov 02 12:59:39.158329 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.158315    5879 cpu_manager.go:414] "containerMap: removing container" podUID="05a0193f-86ca-4a9c-aa04-fd0d80a1743b" containerName="revoke-keys"
Nov 02 12:59:39.158383 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.158331    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a0193f-86ca-4a9c-aa04-fd0d80a1743b" containerName="revoke-keys"
Nov 02 12:59:39.158383 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.158346    5879 cpu_manager.go:414] "containerMap: removing container" podUID="66efff16-b760-4e92-89de-c51ed4a48087" containerName="istio-proxy"
Nov 02 12:59:39.158383 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.158352    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66efff16-b760-4e92-89de-c51ed4a48087" containerName="istio-proxy"
Nov 02 12:59:39.158383 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.158371    5879 cpu_manager.go:414] "containerMap: removing container" podUID="572bcf39-cd03-4a12-a5dc-a2e8dca2d872" containerName="revoke-keys"
Nov 02 12:59:39.158383 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.158376    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572bcf39-cd03-4a12-a5dc-a2e8dca2d872" containerName="revoke-keys"
Nov 02 12:59:39.158560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.158402    5879 cpu_manager.go:414] "containerMap: removing container" podUID="280bc232-ca40-4246-ac92-5cda8dac9930" containerName="discovery"
Nov 02 12:59:39.158560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.158407    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280bc232-ca40-4246-ac92-5cda8dac9930" containerName="discovery"
Nov 02 12:59:39.158560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.158494    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="280bc232-ca40-4246-ac92-5cda8dac9930" containerName="discovery"
Nov 02 12:59:39.158560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.158508    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a0193f-86ca-4a9c-aa04-fd0d80a1743b" containerName="revoke-keys"
Nov 02 12:59:39.158560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.158515    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="66efff16-b760-4e92-89de-c51ed4a48087" containerName="istio-proxy"
Nov 02 12:59:39.158560 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.158525    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="572bcf39-cd03-4a12-a5dc-a2e8dca2d872" containerName="revoke-keys"
Nov 02 12:59:39.163855 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.163830    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:39.183631 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.183431    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"]
Nov 02 12:59:39.193136 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.193107    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:59:39.193327 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.193193    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:59:39.252702 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.252664    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2rqr\" (UniqueName: \"kubernetes.io/projected/0cabd776-2d6c-40a2-8abb-661ace6f4de3-kube-api-access-v2rqr\") pod \"maas-api-revoke-sub-c831876c-ghbcm\" (UID: \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:39.252885 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.252720    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c831876c-ghbcm\" (UID: \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:39.252929 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.252890    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c831876c-ghbcm\" (UID: \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:39.354646 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.354617    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2rqr\" (UniqueName: \"kubernetes.io/projected/0cabd776-2d6c-40a2-8abb-661ace6f4de3-kube-api-access-v2rqr\") pod \"maas-api-revoke-sub-c831876c-ghbcm\" (UID: \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:39.354797 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.354673    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c831876c-ghbcm\" (UID: \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:39.354797 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.354777    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c831876c-ghbcm\" (UID: \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:39.355517 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.355493    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c831876c-ghbcm\" (UID: \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:39.358824 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.358800    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c831876c-ghbcm\" (UID: \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:39.380157 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.380125    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2rqr\" (UniqueName: \"kubernetes.io/projected/0cabd776-2d6c-40a2-8abb-661ace6f4de3-kube-api-access-v2rqr\") pod \"maas-api-revoke-sub-c831876c-ghbcm\" (UID: \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:39.478480 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.478446    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:39.613017 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.612911    5879 generic.go:356] "Generic (PLEG): container finished" podID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerID="673e0159cf29ef458c209aedd01897298a6cfa0f0986d64c7ee257f83577016c" exitCode=0
Nov 02 12:59:39.613420 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.613305    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" event={"ID":"752a2b49-f48e-4c28-8595-8ed0a708b928","Type":"ContainerDied","Data":"673e0159cf29ef458c209aedd01897298a6cfa0f0986d64c7ee257f83577016c"}
Nov 02 12:59:39.613420 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.613374    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r" event={"ID":"752a2b49-f48e-4c28-8595-8ed0a708b928","Type":"ContainerDied","Data":"6b307064f69d1175f26de1bed1a493027531c3f8d0c0a4e6ebeea423bb61f3d2"}
Nov 02 12:59:39.613420 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.613401    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b307064f69d1175f26de1bed1a493027531c3f8d0c0a4e6ebeea423bb61f3d2"
Nov 02 12:59:39.620883 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.620855    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:59:39.659124 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659096    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-token\" (UniqueName: \"kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-token\") pod \"752a2b49-f48e-4c28-8595-8ed0a708b928\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") "
Nov 02 12:59:39.659294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659158    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752a2b49-f48e-4c28-8595-8ed0a708b928-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752a2b49-f48e-4c28-8595-8ed0a708b928-istiod-ca-cert\") pod \"752a2b49-f48e-4c28-8595-8ed0a708b928\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") "
Nov 02 12:59:39.659294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659181    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-socket\") pod \"752a2b49-f48e-4c28-8595-8ed0a708b928\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") "
Nov 02 12:59:39.659294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659240    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-podinfo\") pod \"752a2b49-f48e-4c28-8595-8ed0a708b928\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") "
Nov 02 12:59:39.659294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659288    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-data\") pod \"752a2b49-f48e-4c28-8595-8ed0a708b928\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") "
Nov 02 12:59:39.659476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659384    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-envoy\") pod \"752a2b49-f48e-4c28-8595-8ed0a708b928\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") "
Nov 02 12:59:39.659476 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659455    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-credential-socket\") pod \"752a2b49-f48e-4c28-8595-8ed0a708b928\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") "
Nov 02 12:59:39.659596 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659515    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-kube-api-access-8276q\" (UniqueName: \"kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-kube-api-access-8276q\") pod \"752a2b49-f48e-4c28-8595-8ed0a708b928\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") "
Nov 02 12:59:39.659596 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659536    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-socket" pod "752a2b49-f48e-4c28-8595-8ed0a708b928" (UID: "752a2b49-f48e-4c28-8595-8ed0a708b928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:39.659714 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659604    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-certs\") pod \"752a2b49-f48e-4c28-8595-8ed0a708b928\" (UID: \"752a2b49-f48e-4c28-8595-8ed0a708b928\") "
Nov 02 12:59:39.659714 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659633    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-data" pod "752a2b49-f48e-4c28-8595-8ed0a708b928" (UID: "752a2b49-f48e-4c28-8595-8ed0a708b928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:39.659714 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659639    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752a2b49-f48e-4c28-8595-8ed0a708b928-istiod-ca-cert" pod "752a2b49-f48e-4c28-8595-8ed0a708b928" (UID: "752a2b49-f48e-4c28-8595-8ed0a708b928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:39.659879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659760    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-credential-socket" pod "752a2b49-f48e-4c28-8595-8ed0a708b928" (UID: "752a2b49-f48e-4c28-8595-8ed0a708b928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:39.659964 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.659940    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-certs" pod "752a2b49-f48e-4c28-8595-8ed0a708b928" (UID: "752a2b49-f48e-4c28-8595-8ed0a708b928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:39.660610 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.660533    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/752a2b49-f48e-4c28-8595-8ed0a708b928-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.660610 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.660561    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.660610 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.660576    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.660610 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.660593    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.660610 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.660609    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.661785 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.661752    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-envoy" pod "752a2b49-f48e-4c28-8595-8ed0a708b928" (UID: "752a2b49-f48e-4c28-8595-8ed0a708b928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:39.661879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.661793    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-podinfo" pod "752a2b49-f48e-4c28-8595-8ed0a708b928" (UID: "752a2b49-f48e-4c28-8595-8ed0a708b928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:39.661879 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.661825    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-token" pod "752a2b49-f48e-4c28-8595-8ed0a708b928" (UID: "752a2b49-f48e-4c28-8595-8ed0a708b928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:39.662102 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.662085    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-kube-api-access-8276q" pod "752a2b49-f48e-4c28-8595-8ed0a708b928" (UID: "752a2b49-f48e-4c28-8595-8ed0a708b928"). InnerVolumeSpecName "kube-api-access-8276q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:39.761692 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.761617    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.761692 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.761643    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8276q\" (UniqueName: \"kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-kube-api-access-8276q\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.761692 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.761654    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.761692 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.761662    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/752a2b49-f48e-4c28-8595-8ed0a708b928-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:39.812738 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:59:39.812694    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cabd776_2d6c_40a2_8abb_661ace6f4de3.slice/crio-e03ffb3c7e60aab542b8e5c22f65f084c5b4edfdfcbe8d9ab88f9211299bb3cb WatchSource:0}: Error finding container e03ffb3c7e60aab542b8e5c22f65f084c5b4edfdfcbe8d9ab88f9211299bb3cb: Status 404 returned error can't find the container with id e03ffb3c7e60aab542b8e5c22f65f084c5b4edfdfcbe8d9ab88f9211299bb3cb
Nov 02 12:59:39.812878 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:39.812857    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"]
Nov 02 12:59:40.241949 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:40.241920    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:40.387206 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:40.387171    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"]
Nov 02 12:59:40.387441 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:40.387417    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy" containerID="cri-o://4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a" gracePeriod=30
Nov 02 12:59:40.622680 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:40.622643    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm" event={"ID":"0cabd776-2d6c-40a2-8abb-661ace6f4de3","Type":"ContainerStarted","Data":"c75cc10bd782e79136db2849048795b5566bc126cbbc93dfac512ee867c043b7"}
Nov 02 12:59:40.622865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:40.622710    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm" event={"ID":"0cabd776-2d6c-40a2-8abb-661ace6f4de3","Type":"ContainerStarted","Data":"e03ffb3c7e60aab542b8e5c22f65f084c5b4edfdfcbe8d9ab88f9211299bb3cb"}
Nov 02 12:59:40.622865 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:40.622736    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"
Nov 02 12:59:40.668014 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:40.667947    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm" podStartSLOduration=1.667933309 podStartE2EDuration="1.667933309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:59:40.666071188 +0000 UTC m=+1926.772774039" watchObservedRunningTime="2026-11-02 12:59:40.667933309 +0000 UTC m=+1926.774636174"
Nov 02 12:59:40.700445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:40.700391    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:40.707018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:40.706975    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"]
Nov 02 12:59:40.710983 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:40.710957    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-c9d8r"]
Nov 02 12:59:41.363266 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.363227    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"]
Nov 02 12:59:41.364069 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.364048    5879 cpu_manager.go:414] "containerMap: removing container" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerName="istio-proxy"
Nov 02 12:59:41.364134 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.364071    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerName="istio-proxy"
Nov 02 12:59:41.364202 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.364191    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" containerName="istio-proxy"
Nov 02 12:59:41.367730 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.367713    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:41.386886 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.386856    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"]
Nov 02 12:59:41.483698 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.483664    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2960e17-dcx27\" (UID: \"9e016209-454b-42ef-9a94-ad72659a1b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:41.483878 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.483836    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j2pf\" (UniqueName: \"kubernetes.io/projected/9e016209-454b-42ef-9a94-ad72659a1b6a-kube-api-access-9j2pf\") pod \"maas-api-revoke-sub-e2960e17-dcx27\" (UID: \"9e016209-454b-42ef-9a94-ad72659a1b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:41.484069 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.484051    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2960e17-dcx27\" (UID: \"9e016209-454b-42ef-9a94-ad72659a1b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:41.585744 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.585713    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2960e17-dcx27\" (UID: \"9e016209-454b-42ef-9a94-ad72659a1b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:41.585914 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.585794    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2960e17-dcx27\" (UID: \"9e016209-454b-42ef-9a94-ad72659a1b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:41.585969 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.585920    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j2pf\" (UniqueName: \"kubernetes.io/projected/9e016209-454b-42ef-9a94-ad72659a1b6a-kube-api-access-9j2pf\") pod \"maas-api-revoke-sub-e2960e17-dcx27\" (UID: \"9e016209-454b-42ef-9a94-ad72659a1b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:41.586445 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.586418    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2960e17-dcx27\" (UID: \"9e016209-454b-42ef-9a94-ad72659a1b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:41.588675 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.588653    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2960e17-dcx27\" (UID: \"9e016209-454b-42ef-9a94-ad72659a1b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:41.611074 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.611043    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j2pf\" (UniqueName: \"kubernetes.io/projected/9e016209-454b-42ef-9a94-ad72659a1b6a-kube-api-access-9j2pf\") pod \"maas-api-revoke-sub-e2960e17-dcx27\" (UID: \"9e016209-454b-42ef-9a94-ad72659a1b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:41.630619 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.630590    5879 generic.go:356] "Generic (PLEG): container finished" podID="0cabd776-2d6c-40a2-8abb-661ace6f4de3" containerID="c75cc10bd782e79136db2849048795b5566bc126cbbc93dfac512ee867c043b7" exitCode=0
Nov 02 12:59:41.630806 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.630694    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm" event={"ID":"0cabd776-2d6c-40a2-8abb-661ace6f4de3","Type":"ContainerDied","Data":"c75cc10bd782e79136db2849048795b5566bc126cbbc93dfac512ee867c043b7"}
Nov 02 12:59:41.682868 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.682835    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:41.912915 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:41.912816    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:42.002814 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.002781    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"]
Nov 02 12:59:42.006479 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:59:42.006449    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e016209_454b_42ef_9a94_ad72659a1b6a.slice/crio-1971be559cd6272c12302455e58fd4b93974219b2b00630d23a786457fd2f83c WatchSource:0}: Error finding container 1971be559cd6272c12302455e58fd4b93974219b2b00630d23a786457fd2f83c: Status 404 returned error can't find the container with id 1971be559cd6272c12302455e58fd4b93974219b2b00630d23a786457fd2f83c
Nov 02 12:59:42.018087 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.018064    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752a2b49-f48e-4c28-8595-8ed0a708b928" path="/var/lib/kubelet/pods/752a2b49-f48e-4c28-8595-8ed0a708b928/volumes"
Nov 02 12:59:42.639749 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.639712    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27" event={"ID":"9e016209-454b-42ef-9a94-ad72659a1b6a","Type":"ContainerStarted","Data":"00ab4dbbb2edb0b4758b7184036abd3f7f830d90e68281c1c1f140c10d90a008"}
Nov 02 12:59:42.640246 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.639775    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27" event={"ID":"9e016209-454b-42ef-9a94-ad72659a1b6a","Type":"ContainerStarted","Data":"1971be559cd6272c12302455e58fd4b93974219b2b00630d23a786457fd2f83c"}
Nov 02 12:59:42.683666 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.683597    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27" podStartSLOduration=1.683570233 podStartE2EDuration="1.683570233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:59:42.682170146 +0000 UTC m=+1928.788872999" watchObservedRunningTime="2026-11-02 12:59:42.683570233 +0000 UTC m=+1928.790273088"
Nov 02 12:59:42.796421 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.796390    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:42.906558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.906458    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cabd776-2d6c-40a2-8abb-661ace6f4de3-kube-api-access-v2rqr\" (UniqueName: \"kubernetes.io/projected/0cabd776-2d6c-40a2-8abb-661ace6f4de3-kube-api-access-v2rqr\") pod \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\" (UID: \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\") "
Nov 02 12:59:42.906558 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.906552    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-openshift-ca\") pod \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\" (UID: \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\") "
Nov 02 12:59:42.906802 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.906674    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-xks-ca\") pod \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\" (UID: \"0cabd776-2d6c-40a2-8abb-661ace6f4de3\") "
Nov 02 12:59:42.906925 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.906901    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-openshift-ca" pod "0cabd776-2d6c-40a2-8abb-661ace6f4de3" (UID: "0cabd776-2d6c-40a2-8abb-661ace6f4de3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:42.907267 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.907253    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:42.908790 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.908766    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-xks-ca" pod "0cabd776-2d6c-40a2-8abb-661ace6f4de3" (UID: "0cabd776-2d6c-40a2-8abb-661ace6f4de3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:59:42.908908 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:42.908790    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cabd776-2d6c-40a2-8abb-661ace6f4de3-kube-api-access-v2rqr" pod "0cabd776-2d6c-40a2-8abb-661ace6f4de3" (UID: "0cabd776-2d6c-40a2-8abb-661ace6f4de3"). InnerVolumeSpecName "kube-api-access-v2rqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:43.008776 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:43.008743    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cabd776-2d6c-40a2-8abb-661ace6f4de3-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.008776 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:43.008775    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2rqr\" (UniqueName: \"kubernetes.io/projected/0cabd776-2d6c-40a2-8abb-661ace6f4de3-kube-api-access-v2rqr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:43.419285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:43.419247    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:43.649311 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:43.649279    5879 generic.go:356] "Generic (PLEG): container finished" podID="9e016209-454b-42ef-9a94-ad72659a1b6a" containerID="00ab4dbbb2edb0b4758b7184036abd3f7f830d90e68281c1c1f140c10d90a008" exitCode=0
Nov 02 12:59:43.649756 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:43.649367    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27" event={"ID":"9e016209-454b-42ef-9a94-ad72659a1b6a","Type":"ContainerDied","Data":"00ab4dbbb2edb0b4758b7184036abd3f7f830d90e68281c1c1f140c10d90a008"}
Nov 02 12:59:43.651147 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:43.651124    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm" event={"ID":"0cabd776-2d6c-40a2-8abb-661ace6f4de3","Type":"ContainerDied","Data":"e03ffb3c7e60aab542b8e5c22f65f084c5b4edfdfcbe8d9ab88f9211299bb3cb"}
Nov 02 12:59:43.651257 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:43.651161    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e03ffb3c7e60aab542b8e5c22f65f084c5b4edfdfcbe8d9ab88f9211299bb3cb"
Nov 02 12:59:43.651257 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:43.651127    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"
Nov 02 12:59:44.852590 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:59:44.852542    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 12:59:44.859722 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:44.859699    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:44.939589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:44.939493    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-xks-ca\") pod \"9e016209-454b-42ef-9a94-ad72659a1b6a\" (UID: \"9e016209-454b-42ef-9a94-ad72659a1b6a\") "
Nov 02 12:59:44.939589 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:44.939552    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-openshift-ca\") pod \"9e016209-454b-42ef-9a94-ad72659a1b6a\" (UID: \"9e016209-454b-42ef-9a94-ad72659a1b6a\") "
Nov 02 12:59:44.939831 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:44.939654    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e016209-454b-42ef-9a94-ad72659a1b6a-kube-api-access-9j2pf\" (UniqueName: \"kubernetes.io/projected/9e016209-454b-42ef-9a94-ad72659a1b6a-kube-api-access-9j2pf\") pod \"9e016209-454b-42ef-9a94-ad72659a1b6a\" (UID: \"9e016209-454b-42ef-9a94-ad72659a1b6a\") "
Nov 02 12:59:44.940516 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:44.940479    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-openshift-ca" pod "9e016209-454b-42ef-9a94-ad72659a1b6a" (UID: "9e016209-454b-42ef-9a94-ad72659a1b6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:44.942891 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:44.942864    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-xks-ca" pod "9e016209-454b-42ef-9a94-ad72659a1b6a" (UID: "9e016209-454b-42ef-9a94-ad72659a1b6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 12:59:44.943032 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:44.942907    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e016209-454b-42ef-9a94-ad72659a1b6a-kube-api-access-9j2pf" pod "9e016209-454b-42ef-9a94-ad72659a1b6a" (UID: "9e016209-454b-42ef-9a94-ad72659a1b6a"). InnerVolumeSpecName "kube-api-access-9j2pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:45.041777 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.041745    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j2pf\" (UniqueName: \"kubernetes.io/projected/9e016209-454b-42ef-9a94-ad72659a1b6a-kube-api-access-9j2pf\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.041777 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.041777    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.042054 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.041794    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e016209-454b-42ef-9a94-ad72659a1b6a-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.124615 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.124585    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:45.641932 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.641908    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:59:45.667646 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.667556    5879 generic.go:356] "Generic (PLEG): container finished" podID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerID="4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a" exitCode=0
Nov 02 12:59:45.667646 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.667606    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"
Nov 02 12:59:45.667874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.667723    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" event={"ID":"c5191e8a-315a-4d0b-bc4c-692ca33763bd","Type":"ContainerDied","Data":"4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a"}
Nov 02 12:59:45.667874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.667796    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5" event={"ID":"c5191e8a-315a-4d0b-bc4c-692ca33763bd","Type":"ContainerDied","Data":"a28f658ceb3aedb75d60d7f821c208e792ff8181d6f91942bb91b770511d8a9e"}
Nov 02 12:59:45.667874 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.667832    5879 scope.go:122] "RemoveContainer" containerID="4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a"
Nov 02 12:59:45.669484 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.669468    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"
Nov 02 12:59:45.669600 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.669460    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27" event={"ID":"9e016209-454b-42ef-9a94-ad72659a1b6a","Type":"ContainerDied","Data":"1971be559cd6272c12302455e58fd4b93974219b2b00630d23a786457fd2f83c"}
Nov 02 12:59:45.669600 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.669591    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1971be559cd6272c12302455e58fd4b93974219b2b00630d23a786457fd2f83c"
Nov 02 12:59:45.679590 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.679571    5879 scope.go:122] "RemoveContainer" containerID="4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a"
Nov 02 12:59:45.679936 ip-10-0-134-229 kubenswrapper[5879]: E1102 12:59:45.679917    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a\": container with ID starting with 4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a not found: ID does not exist" containerID="4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a"
Nov 02 12:59:45.679982 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.679946    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a"} err="failed to get container status \"4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a\": rpc error: code = NotFound desc = could not find container \"4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a\": container with ID starting with 4bc0dba4a6509797e592ec5012ed206c3d690ad24a8cfb28a657ae92ec80155a not found: ID does not exist"
Nov 02 12:59:45.752296 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752265    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-socket\") pod \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") "
Nov 02 12:59:45.752499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752328    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-credential-socket\") pod \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") "
Nov 02 12:59:45.752499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752347    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-data\") pod \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") "
Nov 02 12:59:45.752499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752407    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-kube-api-access-v86p9\" (UniqueName: \"kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-kube-api-access-v86p9\") pod \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") "
Nov 02 12:59:45.752499 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752451    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-envoy\") pod \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") "
Nov 02 12:59:45.752728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752508    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-token\" (UniqueName: \"kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-token\") pod \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") "
Nov 02 12:59:45.752728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752553    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istiod-ca-cert\") pod \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") "
Nov 02 12:59:45.752728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752596    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-socket" pod "c5191e8a-315a-4d0b-bc4c-692ca33763bd" (UID: "c5191e8a-315a-4d0b-bc4c-692ca33763bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:45.752728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752678    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-podinfo\") pod \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") "
Nov 02 12:59:45.752728 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752704    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-credential-socket" pod "c5191e8a-315a-4d0b-bc4c-692ca33763bd" (UID: "c5191e8a-315a-4d0b-bc4c-692ca33763bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:45.753025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752812    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-certs\") pod \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\" (UID: \"c5191e8a-315a-4d0b-bc4c-692ca33763bd\") "
Nov 02 12:59:45.753025 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752900    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istiod-ca-cert" pod "c5191e8a-315a-4d0b-bc4c-692ca33763bd" (UID: "c5191e8a-315a-4d0b-bc4c-692ca33763bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 12:59:45.753160 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.752988    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-data" pod "c5191e8a-315a-4d0b-bc4c-692ca33763bd" (UID: "c5191e8a-315a-4d0b-bc4c-692ca33763bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:45.753215 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.753167    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-certs" pod "c5191e8a-315a-4d0b-bc4c-692ca33763bd" (UID: "c5191e8a-315a-4d0b-bc4c-692ca33763bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:45.753781 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.753757    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.753892 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.753784    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.753892 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.753799    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.753892 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.753813    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.753892 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.753828    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.755148 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.755124    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-envoy" pod "c5191e8a-315a-4d0b-bc4c-692ca33763bd" (UID: "c5191e8a-315a-4d0b-bc4c-692ca33763bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 12:59:45.755246 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.755181    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-podinfo" pod "c5191e8a-315a-4d0b-bc4c-692ca33763bd" (UID: "c5191e8a-315a-4d0b-bc4c-692ca33763bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 12:59:45.755722 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.755706    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-kube-api-access-v86p9" pod "c5191e8a-315a-4d0b-bc4c-692ca33763bd" (UID: "c5191e8a-315a-4d0b-bc4c-692ca33763bd"). InnerVolumeSpecName "kube-api-access-v86p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:45.755803 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.755723    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-token" pod "c5191e8a-315a-4d0b-bc4c-692ca33763bd" (UID: "c5191e8a-315a-4d0b-bc4c-692ca33763bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 12:59:45.855186 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.855155    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v86p9\" (UniqueName: \"kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-kube-api-access-v86p9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.855186 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.855181    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.855186 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.855189    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:45.855631 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:45.855197    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5191e8a-315a-4d0b-bc4c-692ca33763bd-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 12:59:46.019460 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:46.019423    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"]
Nov 02 12:59:46.033320 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:46.033290    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-6bnj5"]
Nov 02 12:59:46.664479 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:46.664449    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:48.017819 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:48.017785    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" path="/var/lib/kubelet/pods/c5191e8a-315a-4d0b-bc4c-692ca33763bd/volumes"
Nov 02 12:59:48.246336 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:48.246304    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:49.844587 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:49.844553    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:51.438333 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:51.438298    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:53.130598 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:53.130558    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:54.629523 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:54.629489    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:56.366392 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:56.366358    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:57.635146 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:57.633730    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"]
Nov 02 12:59:57.703024 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:57.702960    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:58.069509 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.069405    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:58.192045 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.192018    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:58.231142 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.231105    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"]
Nov 02 12:59:58.232479 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.232452    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy"
Nov 02 12:59:58.232686 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.232636    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy"
Nov 02 12:59:58.232811 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.232799    5879 cpu_manager.go:414] "containerMap: removing container" podUID="0cabd776-2d6c-40a2-8abb-661ace6f4de3" containerName="revoke-keys"
Nov 02 12:59:58.232897 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.232886    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cabd776-2d6c-40a2-8abb-661ace6f4de3" containerName="revoke-keys"
Nov 02 12:59:58.233018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.232986    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9e016209-454b-42ef-9a94-ad72659a1b6a" containerName="revoke-keys"
Nov 02 12:59:58.233109 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.233098    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e016209-454b-42ef-9a94-ad72659a1b6a" containerName="revoke-keys"
Nov 02 12:59:58.233395 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.233379    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5191e8a-315a-4d0b-bc4c-692ca33763bd" containerName="istio-proxy"
Nov 02 12:59:58.233497 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.233486    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cabd776-2d6c-40a2-8abb-661ace6f4de3" containerName="revoke-keys"
Nov 02 12:59:58.233583 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.233572    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e016209-454b-42ef-9a94-ad72659a1b6a" containerName="revoke-keys"
Nov 02 12:59:58.246053 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.244117    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 12:59:58.272029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.269826    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 12:59:58.272029 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.270532    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 12:59:58.276036 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.274405    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"]
Nov 02 12:59:58.343251 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.343158    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdqc8\" (UniqueName: \"kubernetes.io/projected/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-kube-api-access-kdqc8\") pod \"maas-api-revoke-sub-6d5860d8-wkv7q\" (UID: \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 12:59:58.343547 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.343512    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d5860d8-wkv7q\" (UID: \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 12:59:58.343674 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.343585    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d5860d8-wkv7q\" (UID: \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 12:59:58.445408 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.445377    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdqc8\" (UniqueName: \"kubernetes.io/projected/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-kube-api-access-kdqc8\") pod \"maas-api-revoke-sub-6d5860d8-wkv7q\" (UID: \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 12:59:58.445613 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.445450    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d5860d8-wkv7q\" (UID: \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 12:59:58.445613 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.445513    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d5860d8-wkv7q\" (UID: \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 12:59:58.446319 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.446283    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d5860d8-wkv7q\" (UID: \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 12:59:58.448285 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.448260    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d5860d8-wkv7q\" (UID: \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 12:59:58.470714 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.470686    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdqc8\" (UniqueName: \"kubernetes.io/projected/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-kube-api-access-kdqc8\") pod \"maas-api-revoke-sub-6d5860d8-wkv7q\" (UID: \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 12:59:58.572631 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.572589    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 12:59:58.608757 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.608687    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"]
Nov 02 12:59:58.614405 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.614382    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 12:59:58.651199 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.650949    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"]
Nov 02 12:59:58.754510 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.754472    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42de8aea-95mx5\" (UID: \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 12:59:58.754696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.754640    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvm5\" (UniqueName: \"kubernetes.io/projected/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-kube-api-access-6qvm5\") pod \"maas-api-revoke-sub-42de8aea-95mx5\" (UID: \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 12:59:58.754696 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.754691    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42de8aea-95mx5\" (UID: \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 12:59:58.856345 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.856311    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42de8aea-95mx5\" (UID: \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 12:59:58.856571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.856416    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvm5\" (UniqueName: \"kubernetes.io/projected/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-kube-api-access-6qvm5\") pod \"maas-api-revoke-sub-42de8aea-95mx5\" (UID: \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 12:59:58.856571 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.856449    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42de8aea-95mx5\" (UID: \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 12:59:58.857113 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.857094    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42de8aea-95mx5\" (UID: \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 12:59:58.859018 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.858949    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42de8aea-95mx5\" (UID: \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 12:59:58.881294 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.881228    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"]
Nov 02 12:59:58.884599 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:59:58.884571    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90a6ac9_e592_410a_ae4c_aa4f95e9f596.slice/crio-81d77fe09d92411653dc0d597310eb660ee60ca9b8f0145c5901c8cd5c6534aa WatchSource:0}: Error finding container 81d77fe09d92411653dc0d597310eb660ee60ca9b8f0145c5901c8cd5c6534aa: Status 404 returned error can't find the container with id 81d77fe09d92411653dc0d597310eb660ee60ca9b8f0145c5901c8cd5c6534aa
Nov 02 12:59:58.887477 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.887452    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvm5\" (UniqueName: \"kubernetes.io/projected/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-kube-api-access-6qvm5\") pod \"maas-api-revoke-sub-42de8aea-95mx5\" (UID: \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 12:59:58.945592 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:58.945556    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 12:59:59.253757 ip-10-0-134-229 kubenswrapper[5879]: W1102 12:59:59.253727    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc030f6fc_22c2_4f8d_aa39_4337a17ebd4d.slice/crio-b14b40f7a5137324e6ad8353d6144a41da207013575e729e681d96de7a713422 WatchSource:0}: Error finding container b14b40f7a5137324e6ad8353d6144a41da207013575e729e681d96de7a713422: Status 404 returned error can't find the container with id b14b40f7a5137324e6ad8353d6144a41da207013575e729e681d96de7a713422
Nov 02 12:59:59.254853 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:59.254827    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"]
Nov 02 12:59:59.760171 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:59.760130    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 12:59:59.779946 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:59.779907    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5" event={"ID":"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d","Type":"ContainerStarted","Data":"77838c551780dcd6bf040fe365133f22d984f2c04f783b07c6f6c3c91cc36ddc"}
Nov 02 12:59:59.780187 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:59.780019    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5" event={"ID":"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d","Type":"ContainerStarted","Data":"b14b40f7a5137324e6ad8353d6144a41da207013575e729e681d96de7a713422"}
Nov 02 12:59:59.784341 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:59.784307    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q" event={"ID":"e90a6ac9-e592-410a-ae4c-aa4f95e9f596","Type":"ContainerStarted","Data":"b841218141918cc6aba1a0834881ab0ba160c6e728e36f4bb5952eff5d7942e7"}
Nov 02 12:59:59.784545 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:59.784361    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q" event={"ID":"e90a6ac9-e592-410a-ae4c-aa4f95e9f596","Type":"ContainerStarted","Data":"81d77fe09d92411653dc0d597310eb660ee60ca9b8f0145c5901c8cd5c6534aa"}
Nov 02 12:59:59.826095 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:59.826041    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5" podStartSLOduration=1.825986216 podStartE2EDuration="1.825986216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:59:59.825030922 +0000 UTC m=+1945.931733774" watchObservedRunningTime="2026-11-02 12:59:59.825986216 +0000 UTC m=+1945.932689067"
Nov 02 12:59:59.868010 ip-10-0-134-229 kubenswrapper[5879]: I1102 12:59:59.867947    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q" podStartSLOduration=1.867934366 podStartE2EDuration="1.867934366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 12:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 12:59:59.86790775 +0000 UTC m=+1945.974610602" watchObservedRunningTime="2026-11-02 12:59:59.867934366 +0000 UTC m=+1945.974637217"
Nov 02 13:00:00.150506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.150475    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"]
Nov 02 13:00:00.154870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.154844    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"]
Nov 02 13:00:00.155082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.154986    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"
Nov 02 13:00:00.159042 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.158986    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"]
Nov 02 13:00:00.159181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.159096    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"
Nov 02 13:00:00.163298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.163276    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"]
Nov 02 13:00:00.163422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.163380    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"
Nov 02 13:00:00.167196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.167177    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"
Nov 02 13:00:00.179321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.178703    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"]
Nov 02 13:00:00.190297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.185033    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"]
Nov 02 13:00:00.190297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.188059    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"
Nov 02 13:00:00.194518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.194493    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"]
Nov 02 13:00:00.203399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.203370    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"]
Nov 02 13:00:00.203649 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.203602    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"
Nov 02 13:00:00.203830 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.203816    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"]
Nov 02 13:00:00.205098 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.205070    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"]
Nov 02 13:00:00.209720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.209687    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"]
Nov 02 13:00:00.214987 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.214965    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"
Nov 02 13:00:00.223467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.220212    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"]
Nov 02 13:00:00.224885 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.224860    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"]
Nov 02 13:00:00.238240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.229519    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"]
Nov 02 13:00:00.238240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.234917    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"
Nov 02 13:00:00.251039 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.245178    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"]
Nov 02 13:00:00.251039 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.245247    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"]
Nov 02 13:00:00.278637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.278602    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spkwm\" (UniqueName: \"kubernetes.io/projected/33a653e6-e370-43fe-960e-6810aeeec07c-kube-api-access-spkwm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk\" (UID: \"33a653e6-e370-43fe-960e-6810aeeec07c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"
Nov 02 13:00:00.278809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.278696    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f913d0a-8159-4067-bf4d-293bd88a01e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx\" (UID: \"9f913d0a-8159-4067-bf4d-293bd88a01e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"
Nov 02 13:00:00.278880 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.278857    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc3511c-f74f-493c-b446-6f74e4830594-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9\" (UID: \"2cc3511c-f74f-493c-b446-6f74e4830594\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"
Nov 02 13:00:00.278949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.278931    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx9hn\" (UniqueName: \"kubernetes.io/projected/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-kube-api-access-wx9hn\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s\" (UID: \"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"
Nov 02 13:00:00.279013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.278974    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7swch\" (UniqueName: \"kubernetes.io/projected/cc4e5176-3ca6-4719-8555-77b135b17641-kube-api-access-7swch\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb\" (UID: \"cc4e5176-3ca6-4719-8555-77b135b17641\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"
Nov 02 13:00:00.279069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.279052    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc4e5176-3ca6-4719-8555-77b135b17641-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb\" (UID: \"cc4e5176-3ca6-4719-8555-77b135b17641\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"
Nov 02 13:00:00.279117 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.279108    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s\" (UID: \"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"
Nov 02 13:00:00.279163 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.279147    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a653e6-e370-43fe-960e-6810aeeec07c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk\" (UID: \"33a653e6-e370-43fe-960e-6810aeeec07c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"
Nov 02 13:00:00.279220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.279205    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3505c4-6e81-4604-baf9-b02a5051a537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb\" (UID: \"ee3505c4-6e81-4604-baf9-b02a5051a537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"
Nov 02 13:00:00.279287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.279272    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht4pr\" (UniqueName: \"kubernetes.io/projected/2cc3511c-f74f-493c-b446-6f74e4830594-kube-api-access-ht4pr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9\" (UID: \"2cc3511c-f74f-493c-b446-6f74e4830594\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"
Nov 02 13:00:00.279335 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.279318    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrcnn\" (UniqueName: \"kubernetes.io/projected/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-kube-api-access-rrcnn\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc\" (UID: \"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"
Nov 02 13:00:00.279382 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.279368    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc\" (UID: \"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"
Nov 02 13:00:00.279433 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.279411    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4lz6\" (UniqueName: \"kubernetes.io/projected/9f913d0a-8159-4067-bf4d-293bd88a01e8-kube-api-access-q4lz6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx\" (UID: \"9f913d0a-8159-4067-bf4d-293bd88a01e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"
Nov 02 13:00:00.279894 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.279677    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dw8\" (UniqueName: \"kubernetes.io/projected/ee3505c4-6e81-4604-baf9-b02a5051a537-kube-api-access-w6dw8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb\" (UID: \"ee3505c4-6e81-4604-baf9-b02a5051a537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"
Nov 02 13:00:00.380778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.380686    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f913d0a-8159-4067-bf4d-293bd88a01e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx\" (UID: \"9f913d0a-8159-4067-bf4d-293bd88a01e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"
Nov 02 13:00:00.380778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.380750    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc3511c-f74f-493c-b446-6f74e4830594-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9\" (UID: \"2cc3511c-f74f-493c-b446-6f74e4830594\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"
Nov 02 13:00:00.380778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.380794    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx9hn\" (UniqueName: \"kubernetes.io/projected/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-kube-api-access-wx9hn\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s\" (UID: \"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"
Nov 02 13:00:00.381898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.380836    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7swch\" (UniqueName: \"kubernetes.io/projected/cc4e5176-3ca6-4719-8555-77b135b17641-kube-api-access-7swch\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb\" (UID: \"cc4e5176-3ca6-4719-8555-77b135b17641\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"
Nov 02 13:00:00.381898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.380872    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc4e5176-3ca6-4719-8555-77b135b17641-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb\" (UID: \"cc4e5176-3ca6-4719-8555-77b135b17641\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"
Nov 02 13:00:00.381898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.380956    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s\" (UID: \"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"
Nov 02 13:00:00.381898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.381044    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a653e6-e370-43fe-960e-6810aeeec07c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk\" (UID: \"33a653e6-e370-43fe-960e-6810aeeec07c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"
Nov 02 13:00:00.381898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.381119    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3505c4-6e81-4604-baf9-b02a5051a537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb\" (UID: \"ee3505c4-6e81-4604-baf9-b02a5051a537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"
Nov 02 13:00:00.381898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.381202    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf92ef9-e5de-4ba0-a134-661367499778-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85\" (UID: \"4cf92ef9-e5de-4ba0-a134-661367499778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"
Nov 02 13:00:00.381898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.381237    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht4pr\" (UniqueName: \"kubernetes.io/projected/2cc3511c-f74f-493c-b446-6f74e4830594-kube-api-access-ht4pr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9\" (UID: \"2cc3511c-f74f-493c-b446-6f74e4830594\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"
Nov 02 13:00:00.382327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.381965    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrcnn\" (UniqueName: \"kubernetes.io/projected/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-kube-api-access-rrcnn\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc\" (UID: \"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"
Nov 02 13:00:00.382327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.382015    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc3511c-f74f-493c-b446-6f74e4830594-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9\" (UID: \"2cc3511c-f74f-493c-b446-6f74e4830594\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"
Nov 02 13:00:00.382327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.382054    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc\" (UID: \"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"
Nov 02 13:00:00.382327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.382111    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4lz6\" (UniqueName: \"kubernetes.io/projected/9f913d0a-8159-4067-bf4d-293bd88a01e8-kube-api-access-q4lz6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx\" (UID: \"9f913d0a-8159-4067-bf4d-293bd88a01e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"
Nov 02 13:00:00.382327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.382205    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk448\" (UniqueName: \"kubernetes.io/projected/4cf92ef9-e5de-4ba0-a134-661367499778-kube-api-access-kk448\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85\" (UID: \"4cf92ef9-e5de-4ba0-a134-661367499778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"
Nov 02 13:00:00.382327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.382266    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dw8\" (UniqueName: \"kubernetes.io/projected/ee3505c4-6e81-4604-baf9-b02a5051a537-kube-api-access-w6dw8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb\" (UID: \"ee3505c4-6e81-4604-baf9-b02a5051a537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"
Nov 02 13:00:00.382715 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.382378    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spkwm\" (UniqueName: \"kubernetes.io/projected/33a653e6-e370-43fe-960e-6810aeeec07c-kube-api-access-spkwm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk\" (UID: \"33a653e6-e370-43fe-960e-6810aeeec07c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"
Nov 02 13:00:00.382715 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.382435    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f913d0a-8159-4067-bf4d-293bd88a01e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx\" (UID: \"9f913d0a-8159-4067-bf4d-293bd88a01e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"
Nov 02 13:00:00.382715 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.382469    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a653e6-e370-43fe-960e-6810aeeec07c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk\" (UID: \"33a653e6-e370-43fe-960e-6810aeeec07c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"
Nov 02 13:00:00.382879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.382802    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3505c4-6e81-4604-baf9-b02a5051a537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb\" (UID: \"ee3505c4-6e81-4604-baf9-b02a5051a537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"
Nov 02 13:00:00.382942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.382914    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc4e5176-3ca6-4719-8555-77b135b17641-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb\" (UID: \"cc4e5176-3ca6-4719-8555-77b135b17641\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"
Nov 02 13:00:00.383412 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.383389    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc\" (UID: \"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"
Nov 02 13:00:00.383554 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.383535    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s\" (UID: \"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"
Nov 02 13:00:00.406231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.406195    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7swch\" (UniqueName: \"kubernetes.io/projected/cc4e5176-3ca6-4719-8555-77b135b17641-kube-api-access-7swch\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb\" (UID: \"cc4e5176-3ca6-4719-8555-77b135b17641\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"
Nov 02 13:00:00.406852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.406823    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht4pr\" (UniqueName: \"kubernetes.io/projected/2cc3511c-f74f-493c-b446-6f74e4830594-kube-api-access-ht4pr\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9\" (UID: \"2cc3511c-f74f-493c-b446-6f74e4830594\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"
Nov 02 13:00:00.407172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.407148    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx9hn\" (UniqueName: \"kubernetes.io/projected/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-kube-api-access-wx9hn\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s\" (UID: \"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"
Nov 02 13:00:00.407297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.407271    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dw8\" (UniqueName: \"kubernetes.io/projected/ee3505c4-6e81-4604-baf9-b02a5051a537-kube-api-access-w6dw8\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb\" (UID: \"ee3505c4-6e81-4604-baf9-b02a5051a537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"
Nov 02 13:00:00.407652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.407632    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4lz6\" (UniqueName: \"kubernetes.io/projected/9f913d0a-8159-4067-bf4d-293bd88a01e8-kube-api-access-q4lz6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx\" (UID: \"9f913d0a-8159-4067-bf4d-293bd88a01e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"
Nov 02 13:00:00.407924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.407905    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spkwm\" (UniqueName: \"kubernetes.io/projected/33a653e6-e370-43fe-960e-6810aeeec07c-kube-api-access-spkwm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk\" (UID: \"33a653e6-e370-43fe-960e-6810aeeec07c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"
Nov 02 13:00:00.408314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.408297    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrcnn\" (UniqueName: \"kubernetes.io/projected/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-kube-api-access-rrcnn\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc\" (UID: \"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"
Nov 02 13:00:00.472215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.472183    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"
Nov 02 13:00:00.484453 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.484421    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf92ef9-e5de-4ba0-a134-661367499778-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85\" (UID: \"4cf92ef9-e5de-4ba0-a134-661367499778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"
Nov 02 13:00:00.484620 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.484515    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk448\" (UniqueName: \"kubernetes.io/projected/4cf92ef9-e5de-4ba0-a134-661367499778-kube-api-access-kk448\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85\" (UID: \"4cf92ef9-e5de-4ba0-a134-661367499778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"
Nov 02 13:00:00.485199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.485080    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf92ef9-e5de-4ba0-a134-661367499778-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85\" (UID: \"4cf92ef9-e5de-4ba0-a134-661367499778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"
Nov 02 13:00:00.485274 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.485249    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"
Nov 02 13:00:00.510221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.510187    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk448\" (UniqueName: \"kubernetes.io/projected/4cf92ef9-e5de-4ba0-a134-661367499778-kube-api-access-kk448\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85\" (UID: \"4cf92ef9-e5de-4ba0-a134-661367499778\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"
Nov 02 13:00:00.553582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.553549    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"
Nov 02 13:00:00.569986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.569859    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"
Nov 02 13:00:00.578502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.578477    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"
Nov 02 13:00:00.587814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.587778    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"
Nov 02 13:00:00.599074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.596161    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"
Nov 02 13:00:00.603736 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.603708    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"
Nov 02 13:00:00.800900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.800858    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"]
Nov 02 13:00:00.806725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.806612    5879 generic.go:356] "Generic (PLEG): container finished" podID="c030f6fc-22c2-4f8d-aa39-4337a17ebd4d" containerID="77838c551780dcd6bf040fe365133f22d984f2c04f783b07c6f6c3c91cc36ddc" exitCode=0
Nov 02 13:00:00.808618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.806892    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5" event={"ID":"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d","Type":"ContainerDied","Data":"77838c551780dcd6bf040fe365133f22d984f2c04f783b07c6f6c3c91cc36ddc"}
Nov 02 13:00:00.811067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.811037    5879 generic.go:356] "Generic (PLEG): container finished" podID="e90a6ac9-e592-410a-ae4c-aa4f95e9f596" containerID="b841218141918cc6aba1a0834881ab0ba160c6e728e36f4bb5952eff5d7942e7" exitCode=0
Nov 02 13:00:00.811182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.811142    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q" event={"ID":"e90a6ac9-e592-410a-ae4c-aa4f95e9f596","Type":"ContainerDied","Data":"b841218141918cc6aba1a0834881ab0ba160c6e728e36f4bb5952eff5d7942e7"}
Nov 02 13:00:00.821825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.820653    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"]
Nov 02 13:00:00.835690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.835661    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"]
Nov 02 13:00:00.835915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.835894    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" podUID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" containerName="istio-proxy" containerID="cri-o://b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977" gracePeriod=30
Nov 02 13:00:00.978533 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.978444    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"]
Nov 02 13:00:00.982344 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:00.982317    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a653e6_e370_43fe_960e_6810aeeec07c.slice/crio-1ba07ef473ab9d361eb7fe72fb98f77b704f93002d86f01307a86dc547a15041 WatchSource:0}: Error finding container 1ba07ef473ab9d361eb7fe72fb98f77b704f93002d86f01307a86dc547a15041: Status 404 returned error can't find the container with id 1ba07ef473ab9d361eb7fe72fb98f77b704f93002d86f01307a86dc547a15041
Nov 02 13:00:00.984525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.984373    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"]
Nov 02 13:00:00.997976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:00.997801    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"]
Nov 02 13:00:01.002510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.002483    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"]
Nov 02 13:00:01.097319 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:01.097279    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f8e66d_8dd1_413c_a1c9_0fc2dbb0a450.slice/crio-db03c11f1c5ff2d2f4a4f9563c94a83acd1e5d3d0955f357a66dcb4523173bf2 WatchSource:0}: Error finding container db03c11f1c5ff2d2f4a4f9563c94a83acd1e5d3d0955f357a66dcb4523173bf2: Status 404 returned error can't find the container with id db03c11f1c5ff2d2f4a4f9563c94a83acd1e5d3d0955f357a66dcb4523173bf2
Nov 02 13:00:01.111663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.111636    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"]
Nov 02 13:00:01.119293 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.119267    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"]
Nov 02 13:00:01.190592 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:01.190558    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d038f7_0b52_4fbd_94eb_5d2ee5e78aab.slice/crio-1ca7a38340c08a0d8c49e8c22905d58165b2a33e51ee94f4fba433098472d6e2 WatchSource:0}: Error finding container 1ca7a38340c08a0d8c49e8c22905d58165b2a33e51ee94f4fba433098472d6e2: Status 404 returned error can't find the container with id 1ca7a38340c08a0d8c49e8c22905d58165b2a33e51ee94f4fba433098472d6e2
Nov 02 13:00:01.488895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.488821    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:01.820451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.820416    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb" event={"ID":"cc4e5176-3ca6-4719-8555-77b135b17641","Type":"ContainerStarted","Data":"e04be4f6fad842a44aef1165ac6279d657192716bc5eefa4ce9caff774e7a55a"}
Nov 02 13:00:01.820924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.820480    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb" event={"ID":"cc4e5176-3ca6-4719-8555-77b135b17641","Type":"ContainerStarted","Data":"5943ea5e1a5f79c31d58a09ed2510486b92814049adac9493b9414176c21ff58"}
Nov 02 13:00:01.822284 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.822259    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx" event={"ID":"9f913d0a-8159-4067-bf4d-293bd88a01e8","Type":"ContainerStarted","Data":"15f299e33e4ea970fe09c4b407bf9a9d78996c24fe7c886fd9997ecdfac36fc5"}
Nov 02 13:00:01.822416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.822301    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx" event={"ID":"9f913d0a-8159-4067-bf4d-293bd88a01e8","Type":"ContainerStarted","Data":"6a2c3dbf47488d411c17564f8efe913c40d6d62cfac72c33c318bfe00c6ed06b"}
Nov 02 13:00:01.824045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.824025    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s" event={"ID":"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab","Type":"ContainerStarted","Data":"1b9a64f800914b415be0853bb067be6ba44f453a67d8f772d61b7b937312ae99"}
Nov 02 13:00:01.824175 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.824062    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s" event={"ID":"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab","Type":"ContainerStarted","Data":"1ca7a38340c08a0d8c49e8c22905d58165b2a33e51ee94f4fba433098472d6e2"}
Nov 02 13:00:01.825889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.825860    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc" event={"ID":"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450","Type":"ContainerStarted","Data":"82687a123ed845bf94f087390b524c47d8b48b7c3f376cd26f87b5f301af0e19"}
Nov 02 13:00:01.826029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.825906    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc" event={"ID":"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450","Type":"ContainerStarted","Data":"db03c11f1c5ff2d2f4a4f9563c94a83acd1e5d3d0955f357a66dcb4523173bf2"}
Nov 02 13:00:01.827529 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.827505    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb" event={"ID":"ee3505c4-6e81-4604-baf9-b02a5051a537","Type":"ContainerStarted","Data":"85d6af6dec354416279f9caed4eaa115ee732cd9efaaa6eccd76dac7fe59ce78"}
Nov 02 13:00:01.827642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.827547    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb" event={"ID":"ee3505c4-6e81-4604-baf9-b02a5051a537","Type":"ContainerStarted","Data":"42324c94abf76013743268fe511c63599114f36d1f3eb69026da477d84a4eeea"}
Nov 02 13:00:01.829118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.829094    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9" event={"ID":"2cc3511c-f74f-493c-b446-6f74e4830594","Type":"ContainerStarted","Data":"520a037f957aeea5d099eba9f2df6ccb8cbcb14a28c2d7bb87a1ce9502368184"}
Nov 02 13:00:01.829218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.829137    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9" event={"ID":"2cc3511c-f74f-493c-b446-6f74e4830594","Type":"ContainerStarted","Data":"6b77248c8b1696bf44c3c774c027103a380dc899fc84cabacb06ec8d0c48339d"}
Nov 02 13:00:01.830582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.830562    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk" event={"ID":"33a653e6-e370-43fe-960e-6810aeeec07c","Type":"ContainerStarted","Data":"baef3cba824f329d25c6983ac648c821e143ea00e9b6a3c333cefd46c252c70e"}
Nov 02 13:00:01.830701 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.830596    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk" event={"ID":"33a653e6-e370-43fe-960e-6810aeeec07c","Type":"ContainerStarted","Data":"1ba07ef473ab9d361eb7fe72fb98f77b704f93002d86f01307a86dc547a15041"}
Nov 02 13:00:01.832036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.832013    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85" event={"ID":"4cf92ef9-e5de-4ba0-a134-661367499778","Type":"ContainerStarted","Data":"e7dd42faa292e20f1c598e56c0eb8de109e70aad5c5acbf98020fd8147d5e6f5"}
Nov 02 13:00:01.832138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.832058    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85" event={"ID":"4cf92ef9-e5de-4ba0-a134-661367499778","Type":"ContainerStarted","Data":"884915019e88dc0520d35a91d8ccb4a0a6150a3335209a72bcdbeb59d213ae6d"}
Nov 02 13:00:01.862797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.862740    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb" podStartSLOduration=1.862725661 podStartE2EDuration="1.862725661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:01.862497031 +0000 UTC m=+1947.969199883" watchObservedRunningTime="2026-11-02 13:00:01.862725661 +0000 UTC m=+1947.969428511"
Nov 02 13:00:01.917450 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.911039    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk" podStartSLOduration=1.9110243040000001 podStartE2EDuration="1.911024304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:01.904901214 +0000 UTC m=+1948.011604068" watchObservedRunningTime="2026-11-02 13:00:01.911024304 +0000 UTC m=+1948.017727151"
Nov 02 13:00:01.996254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.996199    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx" podStartSLOduration=1.996183034 podStartE2EDuration="1.996183034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:01.948750296 +0000 UTC m=+1948.055453149" watchObservedRunningTime="2026-11-02 13:00:01.996183034 +0000 UTC m=+1948.102885886"
Nov 02 13:00:01.996722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:01.996688    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb" podStartSLOduration=1.996677555 podStartE2EDuration="1.996677555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:01.992747065 +0000 UTC m=+1948.099449917" watchObservedRunningTime="2026-11-02 13:00:01.996677555 +0000 UTC m=+1948.103380406"
Nov 02 13:00:02.041439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.041378    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc" podStartSLOduration=2.041351193 podStartE2EDuration="2.041351193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:02.037757757 +0000 UTC m=+1948.144460609" watchObservedRunningTime="2026-11-02 13:00:02.041351193 +0000 UTC m=+1948.148054045"
Nov 02 13:00:02.072849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.072826    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 13:00:02.076110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.076087    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 13:00:02.085107 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.082412    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85" podStartSLOduration=2.082393561 podStartE2EDuration="2.082393561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:02.07963434 +0000 UTC m=+1948.186337191" watchObservedRunningTime="2026-11-02 13:00:02.082393561 +0000 UTC m=+1948.189096414"
Nov 02 13:00:02.128338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.128274    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s" podStartSLOduration=2.128254064 podStartE2EDuration="2.128254064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:02.123285441 +0000 UTC m=+1948.229988292" watchObservedRunningTime="2026-11-02 13:00:02.128254064 +0000 UTC m=+1948.234956917"
Nov 02 13:00:02.167801 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.167751    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9" podStartSLOduration=2.167736598 podStartE2EDuration="2.167736598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:02.164716258 +0000 UTC m=+1948.271419120" watchObservedRunningTime="2026-11-02 13:00:02.167736598 +0000 UTC m=+1948.274439449"
Nov 02 13:00:02.218982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.218945    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-kube-api-access-6qvm5\" (UniqueName: \"kubernetes.io/projected/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-kube-api-access-6qvm5\") pod \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\" (UID: \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\") "
Nov 02 13:00:02.223909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.220293    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-xks-ca\") pod \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\" (UID: \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\") "
Nov 02 13:00:02.223909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.220798    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-xks-ca\") pod \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\" (UID: \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\") "
Nov 02 13:00:02.223909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.221316    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-kube-api-access-kdqc8\" (UniqueName: \"kubernetes.io/projected/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-kube-api-access-kdqc8\") pod \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\" (UID: \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\") "
Nov 02 13:00:02.223909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.221457    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-openshift-ca\") pod \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\" (UID: \"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d\") "
Nov 02 13:00:02.223909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.221643    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-openshift-ca\") pod \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\" (UID: \"e90a6ac9-e592-410a-ae4c-aa4f95e9f596\") "
Nov 02 13:00:02.223909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.223676    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-openshift-ca" pod "e90a6ac9-e592-410a-ae4c-aa4f95e9f596" (UID: "e90a6ac9-e592-410a-ae4c-aa4f95e9f596"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:02.223909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.223824    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-openshift-ca" pod "c030f6fc-22c2-4f8d-aa39-4337a17ebd4d" (UID: "c030f6fc-22c2-4f8d-aa39-4337a17ebd4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:02.226043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.226017    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-kube-api-access-6qvm5" pod "c030f6fc-22c2-4f8d-aa39-4337a17ebd4d" (UID: "c030f6fc-22c2-4f8d-aa39-4337a17ebd4d"). InnerVolumeSpecName "kube-api-access-6qvm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:02.230179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.229152    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-xks-ca" pod "e90a6ac9-e592-410a-ae4c-aa4f95e9f596" (UID: "e90a6ac9-e592-410a-ae4c-aa4f95e9f596"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:02.230179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.229161    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-xks-ca" pod "c030f6fc-22c2-4f8d-aa39-4337a17ebd4d" (UID: "c030f6fc-22c2-4f8d-aa39-4337a17ebd4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:02.230179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.229185    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-kube-api-access-kdqc8" pod "e90a6ac9-e592-410a-ae4c-aa4f95e9f596" (UID: "e90a6ac9-e592-410a-ae4c-aa4f95e9f596"). InnerVolumeSpecName "kube-api-access-kdqc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:02.323907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.323820    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:02.323907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.323854    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qvm5\" (UniqueName: \"kubernetes.io/projected/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-kube-api-access-6qvm5\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:02.323907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.323867    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:02.323907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.323876    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:02.323907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.323884    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdqc8\" (UniqueName: \"kubernetes.io/projected/e90a6ac9-e592-410a-ae4c-aa4f95e9f596-kube-api-access-kdqc8\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:02.323907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.323893    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:02.842142 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.842109    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"
Nov 02 13:00:02.842142 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.842126    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5" event={"ID":"c030f6fc-22c2-4f8d-aa39-4337a17ebd4d","Type":"ContainerDied","Data":"b14b40f7a5137324e6ad8353d6144a41da207013575e729e681d96de7a713422"}
Nov 02 13:00:02.842674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.842175    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b14b40f7a5137324e6ad8353d6144a41da207013575e729e681d96de7a713422"
Nov 02 13:00:02.843888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.843861    5879 generic.go:356] "Generic (PLEG): container finished" podID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" containerID="1b9a64f800914b415be0853bb067be6ba44f453a67d8f772d61b7b937312ae99" exitCode=60
Nov 02 13:00:02.844064 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.843898    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s" event={"ID":"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab","Type":"ContainerDied","Data":"1b9a64f800914b415be0853bb067be6ba44f453a67d8f772d61b7b937312ae99"}
Nov 02 13:00:02.844322 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.844301    5879 scope.go:122] "RemoveContainer" containerID="1b9a64f800914b415be0853bb067be6ba44f453a67d8f772d61b7b937312ae99"
Nov 02 13:00:02.845902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.845878    5879 generic.go:356] "Generic (PLEG): container finished" podID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" containerID="82687a123ed845bf94f087390b524c47d8b48b7c3f376cd26f87b5f301af0e19" exitCode=60
Nov 02 13:00:02.846036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.845954    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc" event={"ID":"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450","Type":"ContainerDied","Data":"82687a123ed845bf94f087390b524c47d8b48b7c3f376cd26f87b5f301af0e19"}
Nov 02 13:00:02.846276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.846257    5879 scope.go:122] "RemoveContainer" containerID="82687a123ed845bf94f087390b524c47d8b48b7c3f376cd26f87b5f301af0e19"
Nov 02 13:00:02.848121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.847955    5879 generic.go:356] "Generic (PLEG): container finished" podID="ee3505c4-6e81-4604-baf9-b02a5051a537" containerID="85d6af6dec354416279f9caed4eaa115ee732cd9efaaa6eccd76dac7fe59ce78" exitCode=60
Nov 02 13:00:02.848121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.848034    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb" event={"ID":"ee3505c4-6e81-4604-baf9-b02a5051a537","Type":"ContainerDied","Data":"85d6af6dec354416279f9caed4eaa115ee732cd9efaaa6eccd76dac7fe59ce78"}
Nov 02 13:00:02.848412 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.848363    5879 scope.go:122] "RemoveContainer" containerID="85d6af6dec354416279f9caed4eaa115ee732cd9efaaa6eccd76dac7fe59ce78"
Nov 02 13:00:02.850192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.850162    5879 generic.go:356] "Generic (PLEG): container finished" podID="2cc3511c-f74f-493c-b446-6f74e4830594" containerID="520a037f957aeea5d099eba9f2df6ccb8cbcb14a28c2d7bb87a1ce9502368184" exitCode=60
Nov 02 13:00:02.850303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.850246    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9" event={"ID":"2cc3511c-f74f-493c-b446-6f74e4830594","Type":"ContainerDied","Data":"520a037f957aeea5d099eba9f2df6ccb8cbcb14a28c2d7bb87a1ce9502368184"}
Nov 02 13:00:02.850483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.850467    5879 scope.go:122] "RemoveContainer" containerID="520a037f957aeea5d099eba9f2df6ccb8cbcb14a28c2d7bb87a1ce9502368184"
Nov 02 13:00:02.852210 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.852187    5879 generic.go:356] "Generic (PLEG): container finished" podID="33a653e6-e370-43fe-960e-6810aeeec07c" containerID="baef3cba824f329d25c6983ac648c821e143ea00e9b6a3c333cefd46c252c70e" exitCode=60
Nov 02 13:00:02.852319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.852217    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk" event={"ID":"33a653e6-e370-43fe-960e-6810aeeec07c","Type":"ContainerDied","Data":"baef3cba824f329d25c6983ac648c821e143ea00e9b6a3c333cefd46c252c70e"}
Nov 02 13:00:02.852651 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.852635    5879 scope.go:122] "RemoveContainer" containerID="baef3cba824f329d25c6983ac648c821e143ea00e9b6a3c333cefd46c252c70e"
Nov 02 13:00:02.853949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.853931    5879 generic.go:356] "Generic (PLEG): container finished" podID="4cf92ef9-e5de-4ba0-a134-661367499778" containerID="e7dd42faa292e20f1c598e56c0eb8de109e70aad5c5acbf98020fd8147d5e6f5" exitCode=60
Nov 02 13:00:02.854062 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.854019    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85" event={"ID":"4cf92ef9-e5de-4ba0-a134-661367499778","Type":"ContainerDied","Data":"e7dd42faa292e20f1c598e56c0eb8de109e70aad5c5acbf98020fd8147d5e6f5"}
Nov 02 13:00:02.854260 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.854242    5879 scope.go:122] "RemoveContainer" containerID="e7dd42faa292e20f1c598e56c0eb8de109e70aad5c5acbf98020fd8147d5e6f5"
Nov 02 13:00:02.855828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.855811    5879 generic.go:356] "Generic (PLEG): container finished" podID="cc4e5176-3ca6-4719-8555-77b135b17641" containerID="e04be4f6fad842a44aef1165ac6279d657192716bc5eefa4ce9caff774e7a55a" exitCode=60
Nov 02 13:00:02.855903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.855874    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb" event={"ID":"cc4e5176-3ca6-4719-8555-77b135b17641","Type":"ContainerDied","Data":"e04be4f6fad842a44aef1165ac6279d657192716bc5eefa4ce9caff774e7a55a"}
Nov 02 13:00:02.856193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.856177    5879 scope.go:122] "RemoveContainer" containerID="e04be4f6fad842a44aef1165ac6279d657192716bc5eefa4ce9caff774e7a55a"
Nov 02 13:00:02.860409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.860386    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q" event={"ID":"e90a6ac9-e592-410a-ae4c-aa4f95e9f596","Type":"ContainerDied","Data":"81d77fe09d92411653dc0d597310eb660ee60ca9b8f0145c5901c8cd5c6534aa"}
Nov 02 13:00:02.860501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.860428    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81d77fe09d92411653dc0d597310eb660ee60ca9b8f0145c5901c8cd5c6534aa"
Nov 02 13:00:02.860501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.860477    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"
Nov 02 13:00:02.864658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.864522    5879 generic.go:356] "Generic (PLEG): container finished" podID="9f913d0a-8159-4067-bf4d-293bd88a01e8" containerID="15f299e33e4ea970fe09c4b407bf9a9d78996c24fe7c886fd9997ecdfac36fc5" exitCode=60
Nov 02 13:00:02.864658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.864570    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx" event={"ID":"9f913d0a-8159-4067-bf4d-293bd88a01e8","Type":"ContainerDied","Data":"15f299e33e4ea970fe09c4b407bf9a9d78996c24fe7c886fd9997ecdfac36fc5"}
Nov 02 13:00:02.864972 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:02.864899    5879 scope.go:122] "RemoveContainer" containerID="15f299e33e4ea970fe09c4b407bf9a9d78996c24fe7c886fd9997ecdfac36fc5"
Nov 02 13:00:03.204541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:03.204503    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:03.874728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:03.874687    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9" event={"ID":"2cc3511c-f74f-493c-b446-6f74e4830594","Type":"ContainerStarted","Data":"f1c22b00b13b4d2dd3d7dbe88f39b785390d90eeced3cdbbdd4092a74462c51c"}
Nov 02 13:00:03.877168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:03.877141    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk" event={"ID":"33a653e6-e370-43fe-960e-6810aeeec07c","Type":"ContainerStarted","Data":"886bdd3c684963bf83780064339ae22ce9f1fdf8037775c2086418341c204b12"}
Nov 02 13:00:03.879179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:03.879152    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85" event={"ID":"4cf92ef9-e5de-4ba0-a134-661367499778","Type":"ContainerStarted","Data":"7585d340ff876f7fa8181df50e91f92f6667e28c16683d59265bbfbccc8dd71c"}
Nov 02 13:00:03.886146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:03.881451    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb" event={"ID":"cc4e5176-3ca6-4719-8555-77b135b17641","Type":"ContainerStarted","Data":"cf77ef480e727a9752db188af7a65dcee12615d3317d7f58b7ec995c11d5ab3e"}
Nov 02 13:00:03.886830 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:03.886799    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx" event={"ID":"9f913d0a-8159-4067-bf4d-293bd88a01e8","Type":"ContainerStarted","Data":"7feddeea762566fbdc07c0047313c35ecd3b16a15b08be5ae83483e944e431b1"}
Nov 02 13:00:03.888970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:03.888940    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s" event={"ID":"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab","Type":"ContainerStarted","Data":"190bf3ce61eed6d1b061474465d48a269338e75ae7e0e269a6b01fe2b557aff2"}
Nov 02 13:00:03.893203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:03.893180    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc" event={"ID":"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450","Type":"ContainerStarted","Data":"15815b74c27d0c30b66949c6bed1f0b511712a8c2e1779b7ac968e9c12a3ccbb"}
Nov 02 13:00:03.895091 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:03.895070    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb" event={"ID":"ee3505c4-6e81-4604-baf9-b02a5051a537","Type":"ContainerStarted","Data":"ab8287e83f4fb44777f6e7d483e0758ecace3e0b7da9ee3f03df89858bff7f2b"}
Nov 02 13:00:04.826760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.826677    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:04.907299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.907254    5879 generic.go:356] "Generic (PLEG): container finished" podID="cc4e5176-3ca6-4719-8555-77b135b17641" containerID="cf77ef480e727a9752db188af7a65dcee12615d3317d7f58b7ec995c11d5ab3e" exitCode=60
Nov 02 13:00:04.907746 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.907340    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb" event={"ID":"cc4e5176-3ca6-4719-8555-77b135b17641","Type":"ContainerDied","Data":"cf77ef480e727a9752db188af7a65dcee12615d3317d7f58b7ec995c11d5ab3e"}
Nov 02 13:00:04.907746 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.907412    5879 scope.go:122] "RemoveContainer" containerID="e04be4f6fad842a44aef1165ac6279d657192716bc5eefa4ce9caff774e7a55a"
Nov 02 13:00:04.907746 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.907664    5879 scope.go:122] "RemoveContainer" containerID="cf77ef480e727a9752db188af7a65dcee12615d3317d7f58b7ec995c11d5ab3e"
Nov 02 13:00:04.908080 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:04.908054    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb_odh-ai-gateway-infra(cc4e5176-3ca6-4719-8555-77b135b17641)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb" podUID="cc4e5176-3ca6-4719-8555-77b135b17641"
Nov 02 13:00:04.909635 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.909610    5879 generic.go:356] "Generic (PLEG): container finished" podID="9f913d0a-8159-4067-bf4d-293bd88a01e8" containerID="7feddeea762566fbdc07c0047313c35ecd3b16a15b08be5ae83483e944e431b1" exitCode=60
Nov 02 13:00:04.909762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.909657    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx" event={"ID":"9f913d0a-8159-4067-bf4d-293bd88a01e8","Type":"ContainerDied","Data":"7feddeea762566fbdc07c0047313c35ecd3b16a15b08be5ae83483e944e431b1"}
Nov 02 13:00:04.910034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.910009    5879 scope.go:122] "RemoveContainer" containerID="7feddeea762566fbdc07c0047313c35ecd3b16a15b08be5ae83483e944e431b1"
Nov 02 13:00:04.910363 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:04.910337    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx_odh-ai-gateway-infra(9f913d0a-8159-4067-bf4d-293bd88a01e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8"
Nov 02 13:00:04.911868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.911845    5879 generic.go:356] "Generic (PLEG): container finished" podID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" containerID="190bf3ce61eed6d1b061474465d48a269338e75ae7e0e269a6b01fe2b557aff2" exitCode=60
Nov 02 13:00:04.912014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.911873    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s" event={"ID":"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab","Type":"ContainerDied","Data":"190bf3ce61eed6d1b061474465d48a269338e75ae7e0e269a6b01fe2b557aff2"}
Nov 02 13:00:04.912249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.912232    5879 scope.go:122] "RemoveContainer" containerID="190bf3ce61eed6d1b061474465d48a269338e75ae7e0e269a6b01fe2b557aff2"
Nov 02 13:00:04.912518 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:04.912496    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s_odh-ai-gateway-infra(67d038f7-0b52-4fbd-94eb-5d2ee5e78aab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab"
Nov 02 13:00:04.914515 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.914482    5879 generic.go:356] "Generic (PLEG): container finished" podID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" containerID="15815b74c27d0c30b66949c6bed1f0b511712a8c2e1779b7ac968e9c12a3ccbb" exitCode=60
Nov 02 13:00:04.914645 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.914562    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc" event={"ID":"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450","Type":"ContainerDied","Data":"15815b74c27d0c30b66949c6bed1f0b511712a8c2e1779b7ac968e9c12a3ccbb"}
Nov 02 13:00:04.914853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.914820    5879 scope.go:122] "RemoveContainer" containerID="15815b74c27d0c30b66949c6bed1f0b511712a8c2e1779b7ac968e9c12a3ccbb"
Nov 02 13:00:04.915196 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:04.915172    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc_odh-ai-gateway-infra(51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450"
Nov 02 13:00:04.916835 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.916808    5879 generic.go:356] "Generic (PLEG): container finished" podID="ee3505c4-6e81-4604-baf9-b02a5051a537" containerID="ab8287e83f4fb44777f6e7d483e0758ecace3e0b7da9ee3f03df89858bff7f2b" exitCode=60
Nov 02 13:00:04.916920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.916893    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb" event={"ID":"ee3505c4-6e81-4604-baf9-b02a5051a537","Type":"ContainerDied","Data":"ab8287e83f4fb44777f6e7d483e0758ecace3e0b7da9ee3f03df89858bff7f2b"}
Nov 02 13:00:04.917180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.917148    5879 scope.go:122] "RemoveContainer" containerID="ab8287e83f4fb44777f6e7d483e0758ecace3e0b7da9ee3f03df89858bff7f2b"
Nov 02 13:00:04.917456 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:04.917433    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb_odh-ai-gateway-infra(ee3505c4-6e81-4604-baf9-b02a5051a537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537"
Nov 02 13:00:04.918917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.918897    5879 generic.go:356] "Generic (PLEG): container finished" podID="2cc3511c-f74f-493c-b446-6f74e4830594" containerID="f1c22b00b13b4d2dd3d7dbe88f39b785390d90eeced3cdbbdd4092a74462c51c" exitCode=60
Nov 02 13:00:04.919060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.919012    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9" event={"ID":"2cc3511c-f74f-493c-b446-6f74e4830594","Type":"ContainerDied","Data":"f1c22b00b13b4d2dd3d7dbe88f39b785390d90eeced3cdbbdd4092a74462c51c"}
Nov 02 13:00:04.919339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.919197    5879 scope.go:122] "RemoveContainer" containerID="f1c22b00b13b4d2dd3d7dbe88f39b785390d90eeced3cdbbdd4092a74462c51c"
Nov 02 13:00:04.919583 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:04.919563    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9_odh-ai-gateway-infra(2cc3511c-f74f-493c-b446-6f74e4830594)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9" podUID="2cc3511c-f74f-493c-b446-6f74e4830594"
Nov 02 13:00:04.921122 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.921104    5879 generic.go:356] "Generic (PLEG): container finished" podID="33a653e6-e370-43fe-960e-6810aeeec07c" containerID="886bdd3c684963bf83780064339ae22ce9f1fdf8037775c2086418341c204b12" exitCode=60
Nov 02 13:00:04.921221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.921183    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk" event={"ID":"33a653e6-e370-43fe-960e-6810aeeec07c","Type":"ContainerDied","Data":"886bdd3c684963bf83780064339ae22ce9f1fdf8037775c2086418341c204b12"}
Nov 02 13:00:04.921473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.921456    5879 scope.go:122] "RemoveContainer" containerID="886bdd3c684963bf83780064339ae22ce9f1fdf8037775c2086418341c204b12"
Nov 02 13:00:04.921670 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:04.921653    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk_odh-ai-gateway-infra(33a653e6-e370-43fe-960e-6810aeeec07c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk" podUID="33a653e6-e370-43fe-960e-6810aeeec07c"
Nov 02 13:00:04.924045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.924023    5879 generic.go:356] "Generic (PLEG): container finished" podID="4cf92ef9-e5de-4ba0-a134-661367499778" containerID="7585d340ff876f7fa8181df50e91f92f6667e28c16683d59265bbfbccc8dd71c" exitCode=60
Nov 02 13:00:04.924133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.924065    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85" event={"ID":"4cf92ef9-e5de-4ba0-a134-661367499778","Type":"ContainerDied","Data":"7585d340ff876f7fa8181df50e91f92f6667e28c16683d59265bbfbccc8dd71c"}
Nov 02 13:00:04.924232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.924218    5879 scope.go:122] "RemoveContainer" containerID="7585d340ff876f7fa8181df50e91f92f6667e28c16683d59265bbfbccc8dd71c"
Nov 02 13:00:04.924401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.924382    5879 scope.go:122] "RemoveContainer" containerID="15f299e33e4ea970fe09c4b407bf9a9d78996c24fe7c886fd9997ecdfac36fc5"
Nov 02 13:00:04.924499 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:04.924442    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85_odh-ai-gateway-infra(4cf92ef9-e5de-4ba0-a134-661367499778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85" podUID="4cf92ef9-e5de-4ba0-a134-661367499778"
Nov 02 13:00:04.939071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.939049    5879 scope.go:122] "RemoveContainer" containerID="1b9a64f800914b415be0853bb067be6ba44f453a67d8f772d61b7b937312ae99"
Nov 02 13:00:04.956071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.956049    5879 scope.go:122] "RemoveContainer" containerID="82687a123ed845bf94f087390b524c47d8b48b7c3f376cd26f87b5f301af0e19"
Nov 02 13:00:04.970687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.970662    5879 scope.go:122] "RemoveContainer" containerID="85d6af6dec354416279f9caed4eaa115ee732cd9efaaa6eccd76dac7fe59ce78"
Nov 02 13:00:04.985790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:04.985768    5879 scope.go:122] "RemoveContainer" containerID="520a037f957aeea5d099eba9f2df6ccb8cbcb14a28c2d7bb87a1ce9502368184"
Nov 02 13:00:05.001519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:05.001493    5879 scope.go:122] "RemoveContainer" containerID="baef3cba824f329d25c6983ac648c821e143ea00e9b6a3c333cefd46c252c70e"
Nov 02 13:00:05.016879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:05.016855    5879 scope.go:122] "RemoveContainer" containerID="e7dd42faa292e20f1c598e56c0eb8de109e70aad5c5acbf98020fd8147d5e6f5"
Nov 02 13:00:05.935969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:05.935943    5879 scope.go:122] "RemoveContainer" containerID="190bf3ce61eed6d1b061474465d48a269338e75ae7e0e269a6b01fe2b557aff2"
Nov 02 13:00:05.936431 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:05.936323    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s_odh-ai-gateway-infra(67d038f7-0b52-4fbd-94eb-5d2ee5e78aab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab"
Nov 02 13:00:05.937887 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:05.937861    5879 scope.go:122] "RemoveContainer" containerID="15815b74c27d0c30b66949c6bed1f0b511712a8c2e1779b7ac968e9c12a3ccbb"
Nov 02 13:00:05.938259 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:05.938235    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc_odh-ai-gateway-infra(51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450"
Nov 02 13:00:05.939570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:05.939537    5879 scope.go:122] "RemoveContainer" containerID="ab8287e83f4fb44777f6e7d483e0758ecace3e0b7da9ee3f03df89858bff7f2b"
Nov 02 13:00:05.939903 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:05.939880    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb_odh-ai-gateway-infra(ee3505c4-6e81-4604-baf9-b02a5051a537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537"
Nov 02 13:00:05.941671 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:05.941653    5879 scope.go:122] "RemoveContainer" containerID="f1c22b00b13b4d2dd3d7dbe88f39b785390d90eeced3cdbbdd4092a74462c51c"
Nov 02 13:00:05.941973 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:05.941949    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9_odh-ai-gateway-infra(2cc3511c-f74f-493c-b446-6f74e4830594)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9" podUID="2cc3511c-f74f-493c-b446-6f74e4830594"
Nov 02 13:00:05.943630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:05.943611    5879 scope.go:122] "RemoveContainer" containerID="886bdd3c684963bf83780064339ae22ce9f1fdf8037775c2086418341c204b12"
Nov 02 13:00:05.943892 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:05.943868    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk_odh-ai-gateway-infra(33a653e6-e370-43fe-960e-6810aeeec07c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk" podUID="33a653e6-e370-43fe-960e-6810aeeec07c"
Nov 02 13:00:05.945500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:05.945481    5879 scope.go:122] "RemoveContainer" containerID="7585d340ff876f7fa8181df50e91f92f6667e28c16683d59265bbfbccc8dd71c"
Nov 02 13:00:05.945786 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:05.945759    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85_odh-ai-gateway-infra(4cf92ef9-e5de-4ba0-a134-661367499778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85" podUID="4cf92ef9-e5de-4ba0-a134-661367499778"
Nov 02 13:00:05.947317 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:05.947293    5879 scope.go:122] "RemoveContainer" containerID="cf77ef480e727a9752db188af7a65dcee12615d3317d7f58b7ec995c11d5ab3e"
Nov 02 13:00:05.947640 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:05.947602    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb_odh-ai-gateway-infra(cc4e5176-3ca6-4719-8555-77b135b17641)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb" podUID="cc4e5176-3ca6-4719-8555-77b135b17641"
Nov 02 13:00:05.949250 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:05.949230    5879 scope.go:122] "RemoveContainer" containerID="7feddeea762566fbdc07c0047313c35ecd3b16a15b08be5ae83483e944e431b1"
Nov 02 13:00:05.949607 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:05.949584    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx_odh-ai-gateway-infra(9f913d0a-8159-4067-bf4d-293bd88a01e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8"
Nov 02 13:00:06.137855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.137827    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 13:00:06.184296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.184263    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-data\") pod \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") "
Nov 02 13:00:06.184496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.184309    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-kube-api-access-86zhp\" (UniqueName: \"kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-kube-api-access-86zhp\") pod \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") "
Nov 02 13:00:06.184496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.184381    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-envoy\") pod \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") "
Nov 02 13:00:06.184496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.184415    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-certs\") pod \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") "
Nov 02 13:00:06.184678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.184551    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-podinfo\") pod \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") "
Nov 02 13:00:06.184678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.184634    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-data" pod "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" (UID: "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:06.184678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.184648    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istiod-ca-cert\") pod \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") "
Nov 02 13:00:06.184840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.184684    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-socket\") pod \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") "
Nov 02 13:00:06.184840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.184738    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-credential-socket\") pod \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") "
Nov 02 13:00:06.184840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.184787    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-certs" pod "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" (UID: "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:06.184840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.184820    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-token\" (UniqueName: \"kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-token\") pod \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\" (UID: \"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b\") "
Nov 02 13:00:06.185079 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.185016    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-socket" pod "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" (UID: "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:06.185143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.185103    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-credential-socket" pod "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" (UID: "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:06.185197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.185134    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istiod-ca-cert" pod "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" (UID: "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:06.185534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.185505    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:06.185534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.185534    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:06.185718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.185549    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:06.185718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.185564    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:06.185718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.185578    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:06.187559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.187535    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-podinfo" pod "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" (UID: "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:00:06.187672 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.187562    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-kube-api-access-86zhp" pod "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" (UID: "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b"). InnerVolumeSpecName "kube-api-access-86zhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:06.187733 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.187706    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-envoy" pod "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" (UID: "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:06.187823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.187805    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-token" pod "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" (UID: "3d422059-e2fc-48d1-bc39-d0ed04ab8b0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:06.286820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.286784    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86zhp\" (UniqueName: \"kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-kube-api-access-86zhp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:06.286820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.286815    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:06.286820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.286825    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:06.287091 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.286833    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:06.552640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.552607    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:06.957921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.957890    5879 generic.go:356] "Generic (PLEG): container finished" podID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" containerID="b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977" exitCode=0
Nov 02 13:00:06.958358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.957944    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"
Nov 02 13:00:06.958358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.957980    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" event={"ID":"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b","Type":"ContainerDied","Data":"b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977"}
Nov 02 13:00:06.958358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.958050    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh" event={"ID":"3d422059-e2fc-48d1-bc39-d0ed04ab8b0b","Type":"ContainerDied","Data":"59f556f2c8b893a0706604e518a3cdb96ad76d045d81800b909db6317c26324c"}
Nov 02 13:00:06.958358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.958075    5879 scope.go:122] "RemoveContainer" containerID="b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977"
Nov 02 13:00:06.970175 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.970156    5879 scope.go:122] "RemoveContainer" containerID="b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977"
Nov 02 13:00:06.970524 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:06.970507    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977\": container with ID starting with b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977 not found: ID does not exist" containerID="b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977"
Nov 02 13:00:06.970567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:06.970534    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977"} err="failed to get container status \"b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977\": rpc error: code = NotFound desc = could not find container \"b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977\": container with ID starting with b8f6649b21cdbd5d4a97727274c0ee30aa45f49bcc2c358f9b6d180fe4a49977 not found: ID does not exist"
Nov 02 13:00:07.017889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:07.017847    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"]
Nov 02 13:00:07.020921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:07.020902    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-sl6nh"]
Nov 02 13:00:08.018706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.018671    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" path="/var/lib/kubelet/pods/3d422059-e2fc-48d1-bc39-d0ed04ab8b0b/volumes"
Nov 02 13:00:08.148139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.148112    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:08.701663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.701627    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"]
Nov 02 13:00:08.702329 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.702313    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e90a6ac9-e592-410a-ae4c-aa4f95e9f596" containerName="revoke-keys"
Nov 02 13:00:08.702372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.702331    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90a6ac9-e592-410a-ae4c-aa4f95e9f596" containerName="revoke-keys"
Nov 02 13:00:08.702372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.702347    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c030f6fc-22c2-4f8d-aa39-4337a17ebd4d" containerName="revoke-keys"
Nov 02 13:00:08.702372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.702352    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c030f6fc-22c2-4f8d-aa39-4337a17ebd4d" containerName="revoke-keys"
Nov 02 13:00:08.702480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.702382    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" containerName="istio-proxy"
Nov 02 13:00:08.702480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.702389    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" containerName="istio-proxy"
Nov 02 13:00:08.702543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.702523    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90a6ac9-e592-410a-ae4c-aa4f95e9f596" containerName="revoke-keys"
Nov 02 13:00:08.702543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.702532    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c030f6fc-22c2-4f8d-aa39-4337a17ebd4d" containerName="revoke-keys"
Nov 02 13:00:08.702543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.702542    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d422059-e2fc-48d1-bc39-d0ed04ab8b0b" containerName="istio-proxy"
Nov 02 13:00:08.707570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.707550    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:08.720213 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.720176    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a4812e-5xfd8\" (UID: \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:08.720333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.720321    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6vvl\" (UniqueName: \"kubernetes.io/projected/1082fd85-ff1f-4183-ba9f-a0549c57adc2-kube-api-access-d6vvl\") pod \"maas-api-revoke-sub-a5a4812e-5xfd8\" (UID: \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:08.720444 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.720425    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a4812e-5xfd8\" (UID: \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:08.727009 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.726967    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"]
Nov 02 13:00:08.736506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.736481    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:00:08.822119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.822088    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a4812e-5xfd8\" (UID: \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:08.822314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.822173    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6vvl\" (UniqueName: \"kubernetes.io/projected/1082fd85-ff1f-4183-ba9f-a0549c57adc2-kube-api-access-d6vvl\") pod \"maas-api-revoke-sub-a5a4812e-5xfd8\" (UID: \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:08.822314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.822235    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a4812e-5xfd8\" (UID: \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:08.822739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.822709    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a4812e-5xfd8\" (UID: \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:08.824797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.824766    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a4812e-5xfd8\" (UID: \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:08.846938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:08.846912    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6vvl\" (UniqueName: \"kubernetes.io/projected/1082fd85-ff1f-4183-ba9f-a0549c57adc2-kube-api-access-d6vvl\") pod \"maas-api-revoke-sub-a5a4812e-5xfd8\" (UID: \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:09.021979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.021948    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:09.039435 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.039412    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"]
Nov 02 13:00:09.044941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.044918    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:09.064153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.064104    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"]
Nov 02 13:00:09.126421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.126379    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69w4d\" (UniqueName: \"kubernetes.io/projected/f256cf5a-c275-4b6a-b0ae-8af247ad5355-kube-api-access-69w4d\") pod \"maas-api-revoke-sub-e378974b-45dwr\" (UID: \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:09.126581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.126432    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e378974b-45dwr\" (UID: \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:09.126716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.126689    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e378974b-45dwr\" (UID: \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:09.228172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.228084    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69w4d\" (UniqueName: \"kubernetes.io/projected/f256cf5a-c275-4b6a-b0ae-8af247ad5355-kube-api-access-69w4d\") pod \"maas-api-revoke-sub-e378974b-45dwr\" (UID: \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:09.228172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.228120    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e378974b-45dwr\" (UID: \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:09.228421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.228204    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e378974b-45dwr\" (UID: \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:09.228833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.228804    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e378974b-45dwr\" (UID: \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:09.230782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.230760    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e378974b-45dwr\" (UID: \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:09.252865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.252833    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69w4d\" (UniqueName: \"kubernetes.io/projected/f256cf5a-c275-4b6a-b0ae-8af247ad5355-kube-api-access-69w4d\") pod \"maas-api-revoke-sub-e378974b-45dwr\" (UID: \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:09.338683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.338651    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"]
Nov 02 13:00:09.338958 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:09.338934    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1082fd85_ff1f_4183_ba9f_a0549c57adc2.slice/crio-5db81b7111aeced5d5c5f4f5284cf01a03a7e5bcddd898b4642cada8e375ef35 WatchSource:0}: Error finding container 5db81b7111aeced5d5c5f4f5284cf01a03a7e5bcddd898b4642cada8e375ef35: Status 404 returned error can't find the container with id 5db81b7111aeced5d5c5f4f5284cf01a03a7e5bcddd898b4642cada8e375ef35
Nov 02 13:00:09.377933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.377903    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:09.684926 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:09.684877    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf256cf5a_c275_4b6a_b0ae_8af247ad5355.slice/crio-90df48ea7fe8f15e8f9fc7d59013ad81f89ef94059ef6870e7963e1431eaa151 WatchSource:0}: Error finding container 90df48ea7fe8f15e8f9fc7d59013ad81f89ef94059ef6870e7963e1431eaa151: Status 404 returned error can't find the container with id 90df48ea7fe8f15e8f9fc7d59013ad81f89ef94059ef6870e7963e1431eaa151
Nov 02 13:00:09.685230 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.685206    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"]
Nov 02 13:00:09.866662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.866630    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:09.990317 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.990274    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8" event={"ID":"1082fd85-ff1f-4183-ba9f-a0549c57adc2","Type":"ContainerStarted","Data":"705ea9e08c8413ab96993364fa12399073e9ba1d928123807909c487586d825b"}
Nov 02 13:00:09.990459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.990349    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8" event={"ID":"1082fd85-ff1f-4183-ba9f-a0549c57adc2","Type":"ContainerStarted","Data":"5db81b7111aeced5d5c5f4f5284cf01a03a7e5bcddd898b4642cada8e375ef35"}
Nov 02 13:00:09.993722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:09.993460    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr" event={"ID":"f256cf5a-c275-4b6a-b0ae-8af247ad5355","Type":"ContainerStarted","Data":"90df48ea7fe8f15e8f9fc7d59013ad81f89ef94059ef6870e7963e1431eaa151"}
Nov 02 13:00:10.040982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.040926    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8" podStartSLOduration=2.040912605 podStartE2EDuration="2.040912605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:10.037813231 +0000 UTC m=+1956.144516083" watchObservedRunningTime="2026-11-02 13:00:10.040912605 +0000 UTC m=+1956.147615475"
Nov 02 13:00:10.043072 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.043045    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:10.228478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.228443    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7001-b0bffd"]
Nov 02 13:00:10.233661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.233631    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-b0bffd"
Nov 02 13:00:10.250606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.250573    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hfcns\""
Nov 02 13:00:10.250606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.250579    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7001-b0bffd"]
Nov 02 13:00:10.347029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.344077    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4std\" (UniqueName: \"kubernetes.io/projected/89f351fb-4f00-485a-b6d1-e85a83c687ee-kube-api-access-x4std\") pod \"test-curl-7001-b0bffd\" (UID: \"89f351fb-4f00-485a-b6d1-e85a83c687ee\") " pod="opendatahub/test-curl-7001-b0bffd"
Nov 02 13:00:10.430884 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.430846    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:10.445268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.445231    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4std\" (UniqueName: \"kubernetes.io/projected/89f351fb-4f00-485a-b6d1-e85a83c687ee-kube-api-access-x4std\") pod \"test-curl-7001-b0bffd\" (UID: \"89f351fb-4f00-485a-b6d1-e85a83c687ee\") " pod="opendatahub/test-curl-7001-b0bffd"
Nov 02 13:00:10.509885 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.509771    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4std\" (UniqueName: \"kubernetes.io/projected/89f351fb-4f00-485a-b6d1-e85a83c687ee-kube-api-access-x4std\") pod \"test-curl-7001-b0bffd\" (UID: \"89f351fb-4f00-485a-b6d1-e85a83c687ee\") " pod="opendatahub/test-curl-7001-b0bffd"
Nov 02 13:00:10.548817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.548780    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-b0bffd"
Nov 02 13:00:10.857234 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.857204    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7001-b0bffd"]
Nov 02 13:00:10.859008 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:10.858965    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89f351fb_4f00_485a_b6d1_e85a83c687ee.slice/crio-7f5759d7ba61a20cf4264d1c616902b3f21c21ff65ecc659eeda036290c4ed94 WatchSource:0}: Error finding container 7f5759d7ba61a20cf4264d1c616902b3f21c21ff65ecc659eeda036290c4ed94: Status 404 returned error can't find the container with id 7f5759d7ba61a20cf4264d1c616902b3f21c21ff65ecc659eeda036290c4ed94
Nov 02 13:00:10.861146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:10.861128    5879 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 13:00:11.002416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:11.002379    5879 generic.go:356] "Generic (PLEG): container finished" podID="1082fd85-ff1f-4183-ba9f-a0549c57adc2" containerID="705ea9e08c8413ab96993364fa12399073e9ba1d928123807909c487586d825b" exitCode=0
Nov 02 13:00:11.002598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:11.002472    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8" event={"ID":"1082fd85-ff1f-4183-ba9f-a0549c57adc2","Type":"ContainerDied","Data":"705ea9e08c8413ab96993364fa12399073e9ba1d928123807909c487586d825b"}
Nov 02 13:00:11.004270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:11.004236    5879 generic.go:356] "Generic (PLEG): container finished" podID="f256cf5a-c275-4b6a-b0ae-8af247ad5355" containerID="6a434892c71579d191319df7ea34550934da6d4381dc3e69a6b5f00a8442b1f0" exitCode=0
Nov 02 13:00:11.004402 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:11.004321    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr" event={"ID":"f256cf5a-c275-4b6a-b0ae-8af247ad5355","Type":"ContainerDied","Data":"6a434892c71579d191319df7ea34550934da6d4381dc3e69a6b5f00a8442b1f0"}
Nov 02 13:00:11.005623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:11.005599    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7001-b0bffd" event={"ID":"89f351fb-4f00-485a-b6d1-e85a83c687ee","Type":"ContainerStarted","Data":"7f5759d7ba61a20cf4264d1c616902b3f21c21ff65ecc659eeda036290c4ed94"}
Nov 02 13:00:11.094065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:11.093945    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:11.520431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:11.520390    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:11.655470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:11.655431    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:11.780465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:11.780371    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:12.255834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.255799    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:12.258858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.258839    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:12.373115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.372973    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f256cf5a-c275-4b6a-b0ae-8af247ad5355-kube-api-access-69w4d\" (UniqueName: \"kubernetes.io/projected/f256cf5a-c275-4b6a-b0ae-8af247ad5355-kube-api-access-69w4d\") pod \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\" (UID: \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\") "
Nov 02 13:00:12.373473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.373448    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-openshift-ca\") pod \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\" (UID: \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\") "
Nov 02 13:00:12.373591 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.373528    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-xks-ca\") pod \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\" (UID: \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\") "
Nov 02 13:00:12.373652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.373595    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-openshift-ca\") pod \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\" (UID: \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\") "
Nov 02 13:00:12.373652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.373643    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1082fd85-ff1f-4183-ba9f-a0549c57adc2-kube-api-access-d6vvl\" (UniqueName: \"kubernetes.io/projected/1082fd85-ff1f-4183-ba9f-a0549c57adc2-kube-api-access-d6vvl\") pod \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\" (UID: \"1082fd85-ff1f-4183-ba9f-a0549c57adc2\") "
Nov 02 13:00:12.374095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.373798    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-xks-ca\") pod \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\" (UID: \"f256cf5a-c275-4b6a-b0ae-8af247ad5355\") "
Nov 02 13:00:12.374095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.373879    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-openshift-ca" pod "1082fd85-ff1f-4183-ba9f-a0549c57adc2" (UID: "1082fd85-ff1f-4183-ba9f-a0549c57adc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:12.374284 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.374249    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-openshift-ca" pod "f256cf5a-c275-4b6a-b0ae-8af247ad5355" (UID: "f256cf5a-c275-4b6a-b0ae-8af247ad5355"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:12.374527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.374497    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:12.374624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.374526    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:12.377294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.377270    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-xks-ca" pod "1082fd85-ff1f-4183-ba9f-a0549c57adc2" (UID: "1082fd85-ff1f-4183-ba9f-a0549c57adc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:12.377402 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.377354    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1082fd85-ff1f-4183-ba9f-a0549c57adc2-kube-api-access-d6vvl" pod "1082fd85-ff1f-4183-ba9f-a0549c57adc2" (UID: "1082fd85-ff1f-4183-ba9f-a0549c57adc2"). InnerVolumeSpecName "kube-api-access-d6vvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:12.377534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.377514    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-xks-ca" pod "f256cf5a-c275-4b6a-b0ae-8af247ad5355" (UID: "f256cf5a-c275-4b6a-b0ae-8af247ad5355"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:12.378258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.378232    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f256cf5a-c275-4b6a-b0ae-8af247ad5355-kube-api-access-69w4d" pod "f256cf5a-c275-4b6a-b0ae-8af247ad5355" (UID: "f256cf5a-c275-4b6a-b0ae-8af247ad5355"). InnerVolumeSpecName "kube-api-access-69w4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:12.476639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.476604    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1082fd85-ff1f-4183-ba9f-a0549c57adc2-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:12.476639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.476637    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6vvl\" (UniqueName: \"kubernetes.io/projected/1082fd85-ff1f-4183-ba9f-a0549c57adc2-kube-api-access-d6vvl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:12.476639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.476650    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f256cf5a-c275-4b6a-b0ae-8af247ad5355-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:12.476953 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.476660    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69w4d\" (UniqueName: \"kubernetes.io/projected/f256cf5a-c275-4b6a-b0ae-8af247ad5355-kube-api-access-69w4d\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:12.662440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.662348    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7" podUID="d83a9abe-0c3c-44b2-8961-378aa63ce5ea" containerName="main" containerID="cri-o://cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c" gracePeriod=45
Nov 02 13:00:12.946518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.946487    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 13:00:12.997643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.996386    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tls-certs\") pod \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") "
Nov 02 13:00:12.997643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.996648    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-model-cache\") pod \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") "
Nov 02 13:00:12.997643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.996705    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-kube-api-access-txgm6\" (UniqueName: \"kubernetes.io/projected/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-kube-api-access-txgm6\") pod \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") "
Nov 02 13:00:12.997643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.996788    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-dshm\") pod \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") "
Nov 02 13:00:12.997643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.996820    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-home\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-home\") pod \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") "
Nov 02 13:00:12.997643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.996907    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tmp-dir\") pod \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\" (UID: \"d83a9abe-0c3c-44b2-8961-378aa63ce5ea\") "
Nov 02 13:00:12.997643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.996972    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-model-cache" pod "d83a9abe-0c3c-44b2-8961-378aa63ce5ea" (UID: "d83a9abe-0c3c-44b2-8961-378aa63ce5ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:12.997643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.997347    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-home" pod "d83a9abe-0c3c-44b2-8961-378aa63ce5ea" (UID: "d83a9abe-0c3c-44b2-8961-378aa63ce5ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:12.997643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.997457    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tmp-dir" pod "d83a9abe-0c3c-44b2-8961-378aa63ce5ea" (UID: "d83a9abe-0c3c-44b2-8961-378aa63ce5ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:12.998271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.997714    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:12.998271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.997731    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:12.998271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:12.997743    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:13.000114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.000083    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-dshm" pod "d83a9abe-0c3c-44b2-8961-378aa63ce5ea" (UID: "d83a9abe-0c3c-44b2-8961-378aa63ce5ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:13.000588 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.000566    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-kube-api-access-txgm6" pod "d83a9abe-0c3c-44b2-8961-378aa63ce5ea" (UID: "d83a9abe-0c3c-44b2-8961-378aa63ce5ea"). InnerVolumeSpecName "kube-api-access-txgm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:13.000680 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.000609    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tls-certs" pod "d83a9abe-0c3c-44b2-8961-378aa63ce5ea" (UID: "d83a9abe-0c3c-44b2-8961-378aa63ce5ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:13.025186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.025141    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"
Nov 02 13:00:13.025186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.025159    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8" event={"ID":"1082fd85-ff1f-4183-ba9f-a0549c57adc2","Type":"ContainerDied","Data":"5db81b7111aeced5d5c5f4f5284cf01a03a7e5bcddd898b4642cada8e375ef35"}
Nov 02 13:00:13.025446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.025224    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5db81b7111aeced5d5c5f4f5284cf01a03a7e5bcddd898b4642cada8e375ef35"
Nov 02 13:00:13.027332 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.027309    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"
Nov 02 13:00:13.027503 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.027367    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr" event={"ID":"f256cf5a-c275-4b6a-b0ae-8af247ad5355","Type":"ContainerDied","Data":"90df48ea7fe8f15e8f9fc7d59013ad81f89ef94059ef6870e7963e1431eaa151"}
Nov 02 13:00:13.027503 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.027434    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90df48ea7fe8f15e8f9fc7d59013ad81f89ef94059ef6870e7963e1431eaa151"
Nov 02 13:00:13.029806 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.029701    5879 generic.go:356] "Generic (PLEG): container finished" podID="d83a9abe-0c3c-44b2-8961-378aa63ce5ea" containerID="cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c" exitCode=0
Nov 02 13:00:13.029806 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.029757    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7" event={"ID":"d83a9abe-0c3c-44b2-8961-378aa63ce5ea","Type":"ContainerDied","Data":"cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c"}
Nov 02 13:00:13.030117 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.030048    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7" event={"ID":"d83a9abe-0c3c-44b2-8961-378aa63ce5ea","Type":"ContainerDied","Data":"46acc46ea78aa6c546189ed0aff17043a6e600475497ba84fc309200dc24efe1"}
Nov 02 13:00:13.030211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.030166    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"
Nov 02 13:00:13.030453 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.030097    5879 scope.go:122] "RemoveContainer" containerID="cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c"
Nov 02 13:00:13.053318 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.053200    5879 scope.go:122] "RemoveContainer" containerID="cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c"
Nov 02 13:00:13.056829 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:13.056796    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c\": container with ID starting with cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c not found: ID does not exist" containerID="cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c"
Nov 02 13:00:13.056946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.056839    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c"} err="failed to get container status \"cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c\": rpc error: code = NotFound desc = could not find container \"cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c\": container with ID starting with cf02ae80dde31064252e3af8e7d4d00b9c3e4b56084704dcf94c32280749650c not found: ID does not exist"
Nov 02 13:00:13.099639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.099603    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:13.099639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.099639    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txgm6\" (UniqueName: \"kubernetes.io/projected/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-kube-api-access-txgm6\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:13.099910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.099655    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d83a9abe-0c3c-44b2-8961-378aa63ce5ea-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:13.120262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.120225    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"]
Nov 02 13:00:13.124718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.124682    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e7561a/auth-test-model-e7561a-kserve-6bc7696db5-fk7k7"]
Nov 02 13:00:13.258515 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:13.258429    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:14.020852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:14.020821    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d83a9abe-0c3c-44b2-8961-378aa63ce5ea" path="/var/lib/kubelet/pods/d83a9abe-0c3c-44b2-8961-378aa63ce5ea/volumes"
Nov 02 13:00:14.822533 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:14.822493    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:00:14.884338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:14.884304    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:15.005542 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:15.005509    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:15.069065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:15.065492    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7001-b0bffd" event={"ID":"89f351fb-4f00-485a-b6d1-e85a83c687ee","Type":"ContainerStarted","Data":"aba3389ede58bf67143760650c326aebc2515552182925418726272581971edb"}
Nov 02 13:00:15.110577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:15.110472    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7001-b0bffd" podStartSLOduration=2.012645885 podStartE2EDuration="5.110456031s" totalImagesPullingTime="3.097810146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:15.108628392 +0000 UTC m=+1961.215331244" watchObservedRunningTime="2026-11-02 13:00:15.110456031 +0000 UTC m=+1961.217158882"
Nov 02 13:00:16.009933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.009890    5879 scope.go:122] "RemoveContainer" containerID="7585d340ff876f7fa8181df50e91f92f6667e28c16683d59265bbfbccc8dd71c"
Nov 02 13:00:16.010451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.010034    5879 scope.go:122] "RemoveContainer" containerID="15815b74c27d0c30b66949c6bed1f0b511712a8c2e1779b7ac968e9c12a3ccbb"
Nov 02 13:00:16.010451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.010079    5879 scope.go:122] "RemoveContainer" containerID="cf77ef480e727a9752db188af7a65dcee12615d3317d7f58b7ec995c11d5ab3e"
Nov 02 13:00:16.036436 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.034388    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7001-b0bffd"]
Nov 02 13:00:16.038481 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.038456    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7001-b0bffd"]
Nov 02 13:00:16.199518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.199483    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7001-1a9648"]
Nov 02 13:00:16.200219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.200195    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f256cf5a-c275-4b6a-b0ae-8af247ad5355" containerName="revoke-keys"
Nov 02 13:00:16.200219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.200216    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f256cf5a-c275-4b6a-b0ae-8af247ad5355" containerName="revoke-keys"
Nov 02 13:00:16.200372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.200241    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d83a9abe-0c3c-44b2-8961-378aa63ce5ea" containerName="main"
Nov 02 13:00:16.200372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.200246    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83a9abe-0c3c-44b2-8961-378aa63ce5ea" containerName="main"
Nov 02 13:00:16.200372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.200272    5879 cpu_manager.go:414] "containerMap: removing container" podUID="89f351fb-4f00-485a-b6d1-e85a83c687ee" containerName="test-curl-7001-b0bffd"
Nov 02 13:00:16.200372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.200279    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f351fb-4f00-485a-b6d1-e85a83c687ee" containerName="test-curl-7001-b0bffd"
Nov 02 13:00:16.200372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.200297    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1082fd85-ff1f-4183-ba9f-a0549c57adc2" containerName="revoke-keys"
Nov 02 13:00:16.200372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.200301    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1082fd85-ff1f-4183-ba9f-a0549c57adc2" containerName="revoke-keys"
Nov 02 13:00:16.200625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.200410    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1082fd85-ff1f-4183-ba9f-a0549c57adc2" containerName="revoke-keys"
Nov 02 13:00:16.200625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.200425    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f256cf5a-c275-4b6a-b0ae-8af247ad5355" containerName="revoke-keys"
Nov 02 13:00:16.200625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.200435    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83a9abe-0c3c-44b2-8961-378aa63ce5ea" containerName="main"
Nov 02 13:00:16.200625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.200444    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f351fb-4f00-485a-b6d1-e85a83c687ee" containerName="test-curl-7001-b0bffd"
Nov 02 13:00:16.205905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.205881    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-1a9648"
Nov 02 13:00:16.222324 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.222296    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7001-1a9648"]
Nov 02 13:00:16.345334 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.345299    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxl5g\" (UniqueName: \"kubernetes.io/projected/66ad3b6c-5dda-4ae7-9b22-00bc73a388a3-kube-api-access-gxl5g\") pod \"test-curl-7001-1a9648\" (UID: \"66ad3b6c-5dda-4ae7-9b22-00bc73a388a3\") " pod="opendatahub/test-curl-7001-1a9648"
Nov 02 13:00:16.446904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.446866    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxl5g\" (UniqueName: \"kubernetes.io/projected/66ad3b6c-5dda-4ae7-9b22-00bc73a388a3-kube-api-access-gxl5g\") pod \"test-curl-7001-1a9648\" (UID: \"66ad3b6c-5dda-4ae7-9b22-00bc73a388a3\") " pod="opendatahub/test-curl-7001-1a9648"
Nov 02 13:00:16.474140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.474106    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxl5g\" (UniqueName: \"kubernetes.io/projected/66ad3b6c-5dda-4ae7-9b22-00bc73a388a3-kube-api-access-gxl5g\") pod \"test-curl-7001-1a9648\" (UID: \"66ad3b6c-5dda-4ae7-9b22-00bc73a388a3\") " pod="opendatahub/test-curl-7001-1a9648"
Nov 02 13:00:16.520207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.520141    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-1a9648"
Nov 02 13:00:16.596307 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.596234    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:16.718262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.718189    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:16.734380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.734339    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"]
Nov 02 13:00:16.740308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.740283    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"
Nov 02 13:00:16.757720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.757690    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"]
Nov 02 13:00:16.759070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.759029    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ecec44\""
Nov 02 13:00:16.759242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.759145    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ecec44-dockercfg-hx26q\""
Nov 02 13:00:16.835955 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.835857    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7001-1a9648"]
Nov 02 13:00:16.839594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.839566    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:16.853968 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.853883    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2j85\" (UniqueName: \"kubernetes.io/projected/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-kube-api-access-m2j85\") pod \"payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb\" (UID: \"9bc4e19c-58ab-4a52-8496-c1aea5c588ae\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"
Nov 02 13:00:16.854142 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.854116    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb\" (UID: \"9bc4e19c-58ab-4a52-8496-c1aea5c588ae\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"
Nov 02 13:00:16.956200 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.956050    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2j85\" (UniqueName: \"kubernetes.io/projected/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-kube-api-access-m2j85\") pod \"payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb\" (UID: \"9bc4e19c-58ab-4a52-8496-c1aea5c588ae\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"
Nov 02 13:00:16.956889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.956528    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb\" (UID: \"9bc4e19c-58ab-4a52-8496-c1aea5c588ae\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"
Nov 02 13:00:16.957755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.957720    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb\" (UID: \"9bc4e19c-58ab-4a52-8496-c1aea5c588ae\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"
Nov 02 13:00:16.982221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:16.982122    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2j85\" (UniqueName: \"kubernetes.io/projected/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-kube-api-access-m2j85\") pod \"payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb\" (UID: \"9bc4e19c-58ab-4a52-8496-c1aea5c588ae\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"
Nov 02 13:00:17.010373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:17.010347    5879 scope.go:122] "RemoveContainer" containerID="ab8287e83f4fb44777f6e7d483e0758ecace3e0b7da9ee3f03df89858bff7f2b"
Nov 02 13:00:17.010736 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:17.010465    5879 scope.go:122] "RemoveContainer" containerID="190bf3ce61eed6d1b061474465d48a269338e75ae7e0e269a6b01fe2b557aff2"
Nov 02 13:00:17.057115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:17.057083    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"
Nov 02 13:00:17.088432 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:17.088397    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7001-1a9648" event={"ID":"66ad3b6c-5dda-4ae7-9b22-00bc73a388a3","Type":"ContainerStarted","Data":"4b371006abc65a13ee629b0e9e5cee70aa280ec1d886097a51e154233a049350"}
Nov 02 13:00:17.088596 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:17.088448    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7001-1a9648" event={"ID":"66ad3b6c-5dda-4ae7-9b22-00bc73a388a3","Type":"ContainerStarted","Data":"2ceabf29a378d69ebc34d89be257a55e892960cbc137c8d8a321c4c275d9c562"}
Nov 02 13:00:17.091098 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:17.090816    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc" event={"ID":"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450","Type":"ContainerStarted","Data":"59ae115b1ee524d53e8edb117f0af300c812ad185ee812844216258742a17648"}
Nov 02 13:00:17.093677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:17.093649    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85" event={"ID":"4cf92ef9-e5de-4ba0-a134-661367499778","Type":"ContainerStarted","Data":"617d104217729e3e19051bc559ecacd8efde84b00ae231745dcabafd28dec2ac"}
Nov 02 13:00:17.096126 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:17.096102    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb" event={"ID":"cc4e5176-3ca6-4719-8555-77b135b17641","Type":"ContainerStarted","Data":"b034f0de58aed3530571ccad89bd4b679f11f5076772ab254f53382252757ea8"}
Nov 02 13:00:17.096217 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:17.096199    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7001-b0bffd" podUID="89f351fb-4f00-485a-b6d1-e85a83c687ee" containerName="test-curl-7001-b0bffd" containerID="cri-o://aba3389ede58bf67143760650c326aebc2515552182925418726272581971edb" gracePeriod=2
Nov 02 13:00:17.139583 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:17.139525    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7001-1a9648" podStartSLOduration=1.13950953 podStartE2EDuration="1.13950953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:17.137546915 +0000 UTC m=+1963.244249768" watchObservedRunningTime="2026-11-02 13:00:17.13950953 +0000 UTC m=+1963.246212381"
Nov 02 13:00:17.275179 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:17.275140    5879 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7001-b0bffd\" is forbidden: User \"system:node:ip-10-0-134-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-229.ec2.internal' and this object" podUID="89f351fb-4f00-485a-b6d1-e85a83c687ee" pod="opendatahub/test-curl-7001-b0bffd"
Nov 02 13:00:17.491951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:17.491908    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"]
Nov 02 13:00:17.497687 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:17.497523    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc4e19c_58ab_4a52_8496_c1aea5c588ae.slice/crio-f523aadb9cee467cd3c0f1bcda6ca80910e13a3e266992fd2a46432407b3af37 WatchSource:0}: Error finding container f523aadb9cee467cd3c0f1bcda6ca80910e13a3e266992fd2a46432407b3af37: Status 404 returned error can't find the container with id f523aadb9cee467cd3c0f1bcda6ca80910e13a3e266992fd2a46432407b3af37
Nov 02 13:00:18.010860 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.010072    5879 scope.go:122] "RemoveContainer" containerID="7feddeea762566fbdc07c0047313c35ecd3b16a15b08be5ae83483e944e431b1"
Nov 02 13:00:18.010860 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.010505    5879 scope.go:122] "RemoveContainer" containerID="f1c22b00b13b4d2dd3d7dbe88f39b785390d90eeced3cdbbdd4092a74462c51c"
Nov 02 13:00:18.105394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.105356    5879 generic.go:356] "Generic (PLEG): container finished" podID="4cf92ef9-e5de-4ba0-a134-661367499778" containerID="617d104217729e3e19051bc559ecacd8efde84b00ae231745dcabafd28dec2ac" exitCode=60
Nov 02 13:00:18.105559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.105434    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85" event={"ID":"4cf92ef9-e5de-4ba0-a134-661367499778","Type":"ContainerDied","Data":"617d104217729e3e19051bc559ecacd8efde84b00ae231745dcabafd28dec2ac"}
Nov 02 13:00:18.105559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.105514    5879 scope.go:122] "RemoveContainer" containerID="7585d340ff876f7fa8181df50e91f92f6667e28c16683d59265bbfbccc8dd71c"
Nov 02 13:00:18.105942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.105912    5879 scope.go:122] "RemoveContainer" containerID="617d104217729e3e19051bc559ecacd8efde84b00ae231745dcabafd28dec2ac"
Nov 02 13:00:18.106372 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:18.106347    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85_odh-ai-gateway-infra(4cf92ef9-e5de-4ba0-a134-661367499778)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85" podUID="4cf92ef9-e5de-4ba0-a134-661367499778"
Nov 02 13:00:18.107823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.107803    5879 generic.go:356] "Generic (PLEG): container finished" podID="cc4e5176-3ca6-4719-8555-77b135b17641" containerID="b034f0de58aed3530571ccad89bd4b679f11f5076772ab254f53382252757ea8" exitCode=60
Nov 02 13:00:18.107942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.107872    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb" event={"ID":"cc4e5176-3ca6-4719-8555-77b135b17641","Type":"ContainerDied","Data":"b034f0de58aed3530571ccad89bd4b679f11f5076772ab254f53382252757ea8"}
Nov 02 13:00:18.108330 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.108311    5879 scope.go:122] "RemoveContainer" containerID="b034f0de58aed3530571ccad89bd4b679f11f5076772ab254f53382252757ea8"
Nov 02 13:00:18.108678 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:18.108655    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb_odh-ai-gateway-infra(cc4e5176-3ca6-4719-8555-77b135b17641)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb" podUID="cc4e5176-3ca6-4719-8555-77b135b17641"
Nov 02 13:00:18.110298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.110228    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s" event={"ID":"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab","Type":"ContainerStarted","Data":"afd1ef660ee2feb008b27bb7b65e3d818aa3f8bd4c897423960b44da3765ea13"}
Nov 02 13:00:18.112233 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.112212    5879 generic.go:356] "Generic (PLEG): container finished" podID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" containerID="59ae115b1ee524d53e8edb117f0af300c812ad185ee812844216258742a17648" exitCode=60
Nov 02 13:00:18.112350 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.112284    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc" event={"ID":"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450","Type":"ContainerDied","Data":"59ae115b1ee524d53e8edb117f0af300c812ad185ee812844216258742a17648"}
Nov 02 13:00:18.112615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.112596    5879 scope.go:122] "RemoveContainer" containerID="59ae115b1ee524d53e8edb117f0af300c812ad185ee812844216258742a17648"
Nov 02 13:00:18.112849 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:18.112824    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc_odh-ai-gateway-infra(51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450"
Nov 02 13:00:18.114203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.114182    5879 generic.go:356] "Generic (PLEG): container finished" podID="ee3505c4-6e81-4604-baf9-b02a5051a537" containerID="f890a69dadac97f1d46e466ca8083edf500986fa7beba1f7f8a62f85b9092296" exitCode=60
Nov 02 13:00:18.114305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.114250    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb" event={"ID":"ee3505c4-6e81-4604-baf9-b02a5051a537","Type":"ContainerDied","Data":"f890a69dadac97f1d46e466ca8083edf500986fa7beba1f7f8a62f85b9092296"}
Nov 02 13:00:18.114594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.114577    5879 scope.go:122] "RemoveContainer" containerID="f890a69dadac97f1d46e466ca8083edf500986fa7beba1f7f8a62f85b9092296"
Nov 02 13:00:18.114831 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:18.114811    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb_odh-ai-gateway-infra(ee3505c4-6e81-4604-baf9-b02a5051a537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537"
Nov 02 13:00:18.115899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.115880    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb" event={"ID":"9bc4e19c-58ab-4a52-8496-c1aea5c588ae","Type":"ContainerStarted","Data":"1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43"}
Nov 02 13:00:18.115971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.115922    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb" event={"ID":"9bc4e19c-58ab-4a52-8496-c1aea5c588ae","Type":"ContainerStarted","Data":"f523aadb9cee467cd3c0f1bcda6ca80910e13a3e266992fd2a46432407b3af37"}
Nov 02 13:00:18.116045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.115987    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"
Nov 02 13:00:18.124373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.124345    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7001-1a9648"]
Nov 02 13:00:18.128923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.128898    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7001-1a9648"]
Nov 02 13:00:18.197490 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.197454    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"]
Nov 02 13:00:18.199946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.199899    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb" podStartSLOduration=2.199885182 podStartE2EDuration="2.199885182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:18.193910249 +0000 UTC m=+1964.300613113" watchObservedRunningTime="2026-11-02 13:00:18.199885182 +0000 UTC m=+1964.306588102"
Nov 02 13:00:18.278123 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.278092    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"]
Nov 02 13:00:18.281600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.281575    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7001-111df1"]
Nov 02 13:00:18.282284 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.282268    5879 cpu_manager.go:414] "containerMap: removing container" podUID="66ad3b6c-5dda-4ae7-9b22-00bc73a388a3" containerName="test-curl-7001-1a9648"
Nov 02 13:00:18.282334 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.282286    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ad3b6c-5dda-4ae7-9b22-00bc73a388a3" containerName="test-curl-7001-1a9648"
Nov 02 13:00:18.282407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.282396    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ad3b6c-5dda-4ae7-9b22-00bc73a388a3" containerName="test-curl-7001-1a9648"
Nov 02 13:00:18.285943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.285921    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-111df1"
Nov 02 13:00:18.305613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.305586    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7001-111df1"]
Nov 02 13:00:18.371828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.371803    5879 scope.go:122] "RemoveContainer" containerID="cf77ef480e727a9752db188af7a65dcee12615d3317d7f58b7ec995c11d5ab3e"
Nov 02 13:00:18.376910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.376109    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"]
Nov 02 13:00:18.377969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.377860    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87xr6\" (UniqueName: \"kubernetes.io/projected/198a83f1-cd93-42dc-959d-fd4e8db89db5-kube-api-access-87xr6\") pod \"test-curl-7001-111df1\" (UID: \"198a83f1-cd93-42dc-959d-fd4e8db89db5\") " pod="opendatahub/test-curl-7001-111df1"
Nov 02 13:00:18.403102 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.403081    5879 scope.go:122] "RemoveContainer" containerID="15815b74c27d0c30b66949c6bed1f0b511712a8c2e1779b7ac968e9c12a3ccbb"
Nov 02 13:00:18.421457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.421419    5879 scope.go:122] "RemoveContainer" containerID="ab8287e83f4fb44777f6e7d483e0758ecace3e0b7da9ee3f03df89858bff7f2b"
Nov 02 13:00:18.479310 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.479270    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87xr6\" (UniqueName: \"kubernetes.io/projected/198a83f1-cd93-42dc-959d-fd4e8db89db5-kube-api-access-87xr6\") pod \"test-curl-7001-111df1\" (UID: \"198a83f1-cd93-42dc-959d-fd4e8db89db5\") " pod="opendatahub/test-curl-7001-111df1"
Nov 02 13:00:18.505436 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.505404    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87xr6\" (UniqueName: \"kubernetes.io/projected/198a83f1-cd93-42dc-959d-fd4e8db89db5-kube-api-access-87xr6\") pod \"test-curl-7001-111df1\" (UID: \"198a83f1-cd93-42dc-959d-fd4e8db89db5\") " pod="opendatahub/test-curl-7001-111df1"
Nov 02 13:00:18.695669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.695631    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-111df1"
Nov 02 13:00:18.850272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:18.850239    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:19.002592 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:19.002557    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod198a83f1_cd93_42dc_959d_fd4e8db89db5.slice/crio-0a0b260c2aaae20d9d970dc1a65cc26796abee6f52ca694a5858f7dbd59f8327 WatchSource:0}: Error finding container 0a0b260c2aaae20d9d970dc1a65cc26796abee6f52ca694a5858f7dbd59f8327: Status 404 returned error can't find the container with id 0a0b260c2aaae20d9d970dc1a65cc26796abee6f52ca694a5858f7dbd59f8327
Nov 02 13:00:19.003075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.003050    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7001-111df1"]
Nov 02 13:00:19.009759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.009739    5879 scope.go:122] "RemoveContainer" containerID="886bdd3c684963bf83780064339ae22ce9f1fdf8037775c2086418341c204b12"
Nov 02 13:00:19.037308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.037278    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:19.128863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.128835    5879 generic.go:356] "Generic (PLEG): container finished" podID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" containerID="afd1ef660ee2feb008b27bb7b65e3d818aa3f8bd4c897423960b44da3765ea13" exitCode=60
Nov 02 13:00:19.129033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.128903    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s" event={"ID":"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab","Type":"ContainerDied","Data":"afd1ef660ee2feb008b27bb7b65e3d818aa3f8bd4c897423960b44da3765ea13"}
Nov 02 13:00:19.129033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.128971    5879 scope.go:122] "RemoveContainer" containerID="190bf3ce61eed6d1b061474465d48a269338e75ae7e0e269a6b01fe2b557aff2"
Nov 02 13:00:19.129443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.129428    5879 scope.go:122] "RemoveContainer" containerID="afd1ef660ee2feb008b27bb7b65e3d818aa3f8bd4c897423960b44da3765ea13"
Nov 02 13:00:19.129747 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:19.129729    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s_odh-ai-gateway-infra(67d038f7-0b52-4fbd-94eb-5d2ee5e78aab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab"
Nov 02 13:00:19.141198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.141159    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7001-111df1" event={"ID":"198a83f1-cd93-42dc-959d-fd4e8db89db5","Type":"ContainerStarted","Data":"905b0e9e8fe04e7f73cf1fc757015fdd1263181d98d888131f2973a2361ee530"}
Nov 02 13:00:19.141321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.141205    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7001-111df1" event={"ID":"198a83f1-cd93-42dc-959d-fd4e8db89db5","Type":"ContainerStarted","Data":"0a0b260c2aaae20d9d970dc1a65cc26796abee6f52ca694a5858f7dbd59f8327"}
Nov 02 13:00:19.145747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.145716    5879 generic.go:356] "Generic (PLEG): container finished" podID="2cc3511c-f74f-493c-b446-6f74e4830594" containerID="5875bbc363561282deb99c47418102ea538ae32b6d8770afb8e3d003a6a890d0" exitCode=60
Nov 02 13:00:19.145908 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.145890    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9" event={"ID":"2cc3511c-f74f-493c-b446-6f74e4830594","Type":"ContainerDied","Data":"5875bbc363561282deb99c47418102ea538ae32b6d8770afb8e3d003a6a890d0"}
Nov 02 13:00:19.146408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.146389    5879 scope.go:122] "RemoveContainer" containerID="5875bbc363561282deb99c47418102ea538ae32b6d8770afb8e3d003a6a890d0"
Nov 02 13:00:19.146719 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:19.146702    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9_odh-ai-gateway-infra(2cc3511c-f74f-493c-b446-6f74e4830594)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9" podUID="2cc3511c-f74f-493c-b446-6f74e4830594"
Nov 02 13:00:19.153443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.153415    5879 generic.go:356] "Generic (PLEG): container finished" podID="9f913d0a-8159-4067-bf4d-293bd88a01e8" containerID="bb4281aa497b025f6a297bef53dfc0436fad36fa082c952f193d869d7821e914" exitCode=60
Nov 02 13:00:19.153567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.153495    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx" event={"ID":"9f913d0a-8159-4067-bf4d-293bd88a01e8","Type":"ContainerDied","Data":"bb4281aa497b025f6a297bef53dfc0436fad36fa082c952f193d869d7821e914"}
Nov 02 13:00:19.153891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.153872    5879 scope.go:122] "RemoveContainer" containerID="bb4281aa497b025f6a297bef53dfc0436fad36fa082c952f193d869d7821e914"
Nov 02 13:00:19.154386 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:19.154308    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx_odh-ai-gateway-infra(9f913d0a-8159-4067-bf4d-293bd88a01e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8"
Nov 02 13:00:19.155900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.155873    5879 generic.go:356] "Generic (PLEG): container finished" podID="89f351fb-4f00-485a-b6d1-e85a83c687ee" containerID="aba3389ede58bf67143760650c326aebc2515552182925418726272581971edb" exitCode=137
Nov 02 13:00:19.156081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.156064    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7001-1a9648" podUID="66ad3b6c-5dda-4ae7-9b22-00bc73a388a3" containerName="test-curl-7001-1a9648" containerID="cri-o://4b371006abc65a13ee629b0e9e5cee70aa280ec1d886097a51e154233a049350" gracePeriod=2
Nov 02 13:00:19.191338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.190548    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:19.194585 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:19.194556    5879 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7001-1a9648\" is forbidden: User \"system:node:ip-10-0-134-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-229.ec2.internal' and this object" podUID="66ad3b6c-5dda-4ae7-9b22-00bc73a388a3" pod="opendatahub/test-curl-7001-1a9648"
Nov 02 13:00:19.260695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.260659    5879 scope.go:122] "RemoveContainer" containerID="f1c22b00b13b4d2dd3d7dbe88f39b785390d90eeced3cdbbdd4092a74462c51c"
Nov 02 13:00:19.307957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.298114    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"]
Nov 02 13:00:19.307957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.300461    5879 scope.go:122] "RemoveContainer" containerID="7feddeea762566fbdc07c0047313c35ecd3b16a15b08be5ae83483e944e431b1"
Nov 02 13:00:19.407516 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.407399    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"
Nov 02 13:00:19.413727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.413603    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"]
Nov 02 13:00:19.452103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.452077    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-b0bffd"
Nov 02 13:00:19.468869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.462541    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"
Nov 02 13:00:19.470609 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.470586    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"
Nov 02 13:00:19.494958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.494914    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-kube-api-access-rrcnn\" (UniqueName: \"kubernetes.io/projected/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-kube-api-access-rrcnn\") pod \"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450\" (UID: \"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450\") "
Nov 02 13:00:19.495159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.494965    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-maas-api-service-ca\") pod \"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450\" (UID: \"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450\") "
Nov 02 13:00:19.496129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.496083    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-maas-api-service-ca" pod "51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" (UID: "51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:19.503342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.503316    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-kube-api-access-rrcnn" pod "51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" (UID: "51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450"). InnerVolumeSpecName "kube-api-access-rrcnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:19.503505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.503451    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7001-111df1" podStartSLOduration=1.503431697 podStartE2EDuration="1.503431697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:19.499290224 +0000 UTC m=+1965.605993077" watchObservedRunningTime="2026-11-02 13:00:19.503431697 +0000 UTC m=+1965.610134549"
Nov 02 13:00:19.514931 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:19.514829    5879 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7001-1a9648\" is forbidden: User \"system:node:ip-10-0-134-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-229.ec2.internal' and this object" podUID="66ad3b6c-5dda-4ae7-9b22-00bc73a388a3" pod="opendatahub/test-curl-7001-1a9648"
Nov 02 13:00:19.550489 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.550460    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"
Nov 02 13:00:19.596825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.596786    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4e5176-3ca6-4719-8555-77b135b17641-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc4e5176-3ca6-4719-8555-77b135b17641-maas-api-service-ca\") pod \"cc4e5176-3ca6-4719-8555-77b135b17641\" (UID: \"cc4e5176-3ca6-4719-8555-77b135b17641\") "
Nov 02 13:00:19.597213 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.597196    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf92ef9-e5de-4ba0-a134-661367499778-kube-api-access-kk448\" (UniqueName: \"kubernetes.io/projected/4cf92ef9-e5de-4ba0-a134-661367499778-kube-api-access-kk448\") pod \"4cf92ef9-e5de-4ba0-a134-661367499778\" (UID: \"4cf92ef9-e5de-4ba0-a134-661367499778\") "
Nov 02 13:00:19.597504 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.597473    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f351fb-4f00-485a-b6d1-e85a83c687ee-kube-api-access-x4std\" (UniqueName: \"kubernetes.io/projected/89f351fb-4f00-485a-b6d1-e85a83c687ee-kube-api-access-x4std\") pod \"89f351fb-4f00-485a-b6d1-e85a83c687ee\" (UID: \"89f351fb-4f00-485a-b6d1-e85a83c687ee\") "
Nov 02 13:00:19.597687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.597656    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf92ef9-e5de-4ba0-a134-661367499778-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf92ef9-e5de-4ba0-a134-661367499778-maas-api-service-ca\") pod \"4cf92ef9-e5de-4ba0-a134-661367499778\" (UID: \"4cf92ef9-e5de-4ba0-a134-661367499778\") "
Nov 02 13:00:19.597793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.597780    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4e5176-3ca6-4719-8555-77b135b17641-kube-api-access-7swch\" (UniqueName: \"kubernetes.io/projected/cc4e5176-3ca6-4719-8555-77b135b17641-kube-api-access-7swch\") pod \"cc4e5176-3ca6-4719-8555-77b135b17641\" (UID: \"cc4e5176-3ca6-4719-8555-77b135b17641\") "
Nov 02 13:00:19.598613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.598593    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrcnn\" (UniqueName: \"kubernetes.io/projected/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-kube-api-access-rrcnn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:19.598741 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.598730    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:19.601451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.601417    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4e5176-3ca6-4719-8555-77b135b17641-kube-api-access-7swch" pod "cc4e5176-3ca6-4719-8555-77b135b17641" (UID: "cc4e5176-3ca6-4719-8555-77b135b17641"). InnerVolumeSpecName "kube-api-access-7swch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:19.601716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.601696    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4e5176-3ca6-4719-8555-77b135b17641-maas-api-service-ca" pod "cc4e5176-3ca6-4719-8555-77b135b17641" (UID: "cc4e5176-3ca6-4719-8555-77b135b17641"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:19.603800 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.603772    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf92ef9-e5de-4ba0-a134-661367499778-kube-api-access-kk448" pod "4cf92ef9-e5de-4ba0-a134-661367499778" (UID: "4cf92ef9-e5de-4ba0-a134-661367499778"). InnerVolumeSpecName "kube-api-access-kk448". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:19.606681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.606654    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f351fb-4f00-485a-b6d1-e85a83c687ee-kube-api-access-x4std" pod "89f351fb-4f00-485a-b6d1-e85a83c687ee" (UID: "89f351fb-4f00-485a-b6d1-e85a83c687ee"). InnerVolumeSpecName "kube-api-access-x4std". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:19.606929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.606911    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf92ef9-e5de-4ba0-a134-661367499778-maas-api-service-ca" pod "4cf92ef9-e5de-4ba0-a134-661367499778" (UID: "4cf92ef9-e5de-4ba0-a134-661367499778"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:19.700398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.700364    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3505c4-6e81-4604-baf9-b02a5051a537-maas-api-service-ca" pod "ee3505c4-6e81-4604-baf9-b02a5051a537" (UID: "ee3505c4-6e81-4604-baf9-b02a5051a537"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:19.700613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.699981    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3505c4-6e81-4604-baf9-b02a5051a537-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3505c4-6e81-4604-baf9-b02a5051a537-maas-api-service-ca\") pod \"ee3505c4-6e81-4604-baf9-b02a5051a537\" (UID: \"ee3505c4-6e81-4604-baf9-b02a5051a537\") "
Nov 02 13:00:19.700865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.700845    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3505c4-6e81-4604-baf9-b02a5051a537-kube-api-access-w6dw8\" (UniqueName: \"kubernetes.io/projected/ee3505c4-6e81-4604-baf9-b02a5051a537-kube-api-access-w6dw8\") pod \"ee3505c4-6e81-4604-baf9-b02a5051a537\" (UID: \"ee3505c4-6e81-4604-baf9-b02a5051a537\") "
Nov 02 13:00:19.701742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.701719    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc4e5176-3ca6-4719-8555-77b135b17641-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:19.701863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.701850    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee3505c4-6e81-4604-baf9-b02a5051a537-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:19.701953 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.701941    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk448\" (UniqueName: \"kubernetes.io/projected/4cf92ef9-e5de-4ba0-a134-661367499778-kube-api-access-kk448\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:19.702151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.702137    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4std\" (UniqueName: \"kubernetes.io/projected/89f351fb-4f00-485a-b6d1-e85a83c687ee-kube-api-access-x4std\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:19.702251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.702237    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cf92ef9-e5de-4ba0-a134-661367499778-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:19.702339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.702328    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7swch\" (UniqueName: \"kubernetes.io/projected/cc4e5176-3ca6-4719-8555-77b135b17641-kube-api-access-7swch\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:19.705313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.705287    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3505c4-6e81-4604-baf9-b02a5051a537-kube-api-access-w6dw8" pod "ee3505c4-6e81-4604-baf9-b02a5051a537" (UID: "ee3505c4-6e81-4604-baf9-b02a5051a537"). InnerVolumeSpecName "kube-api-access-w6dw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:19.726977 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:19.726936    5879 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7001-1a9648\" is forbidden: User \"system:node:ip-10-0-134-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-229.ec2.internal' and this object" podUID="66ad3b6c-5dda-4ae7-9b22-00bc73a388a3" pod="opendatahub/test-curl-7001-1a9648"
Nov 02 13:00:19.803873 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.803790    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6dw8\" (UniqueName: \"kubernetes.io/projected/ee3505c4-6e81-4604-baf9-b02a5051a537-kube-api-access-w6dw8\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:19.841356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:19.841329    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:20.025803 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.025769    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f351fb-4f00-485a-b6d1-e85a83c687ee" path="/var/lib/kubelet/pods/89f351fb-4f00-485a-b6d1-e85a83c687ee/volumes"
Nov 02 13:00:20.168483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.168456    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-b0bffd"
Nov 02 13:00:20.168945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.168487    5879 scope.go:122] "RemoveContainer" containerID="aba3389ede58bf67143760650c326aebc2515552182925418726272581971edb"
Nov 02 13:00:20.172801 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.172772    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc" event={"ID":"51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450","Type":"ContainerDied","Data":"db03c11f1c5ff2d2f4a4f9563c94a83acd1e5d3d0955f357a66dcb4523173bf2"}
Nov 02 13:00:20.172801 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.172788    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"
Nov 02 13:00:20.175164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.175136    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb" event={"ID":"ee3505c4-6e81-4604-baf9-b02a5051a537","Type":"ContainerDied","Data":"42324c94abf76013743268fe511c63599114f36d1f3eb69026da477d84a4eeea"}
Nov 02 13:00:20.175348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.175330    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"
Nov 02 13:00:20.179459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.179433    5879 generic.go:356] "Generic (PLEG): container finished" podID="33a653e6-e370-43fe-960e-6810aeeec07c" containerID="b489ed1bea26d11bbfcbf8bd8857b44727b0c983fbbd58e1a484b4c6852f40c5" exitCode=60
Nov 02 13:00:20.179594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.179457    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk" event={"ID":"33a653e6-e370-43fe-960e-6810aeeec07c","Type":"ContainerDied","Data":"b489ed1bea26d11bbfcbf8bd8857b44727b0c983fbbd58e1a484b4c6852f40c5"}
Nov 02 13:00:20.179926 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.179907    5879 scope.go:122] "RemoveContainer" containerID="b489ed1bea26d11bbfcbf8bd8857b44727b0c983fbbd58e1a484b4c6852f40c5"
Nov 02 13:00:20.180275 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:20.180243    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk_odh-ai-gateway-infra(33a653e6-e370-43fe-960e-6810aeeec07c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk" podUID="33a653e6-e370-43fe-960e-6810aeeec07c"
Nov 02 13:00:20.180582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.180540    5879 scope.go:122] "RemoveContainer" containerID="59ae115b1ee524d53e8edb117f0af300c812ad185ee812844216258742a17648"
Nov 02 13:00:20.184861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.182735    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"
Nov 02 13:00:20.184861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.182782    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85" event={"ID":"4cf92ef9-e5de-4ba0-a134-661367499778","Type":"ContainerDied","Data":"884915019e88dc0520d35a91d8ccb4a0a6150a3335209a72bcdbeb59d213ae6d"}
Nov 02 13:00:20.185914 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.185404    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"
Nov 02 13:00:20.185914 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.185491    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb" event={"ID":"cc4e5176-3ca6-4719-8555-77b135b17641","Type":"ContainerDied","Data":"5943ea5e1a5f79c31d58a09ed2510486b92814049adac9493b9414176c21ff58"}
Nov 02 13:00:20.193401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.193371    5879 scope.go:122] "RemoveContainer" containerID="f890a69dadac97f1d46e466ca8083edf500986fa7beba1f7f8a62f85b9092296"
Nov 02 13:00:20.204104 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.204078    5879 scope.go:122] "RemoveContainer" containerID="886bdd3c684963bf83780064339ae22ce9f1fdf8037775c2086418341c204b12"
Nov 02 13:00:20.220035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.219980    5879 scope.go:122] "RemoveContainer" containerID="617d104217729e3e19051bc559ecacd8efde84b00ae231745dcabafd28dec2ac"
Nov 02 13:00:20.232082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.232059    5879 scope.go:122] "RemoveContainer" containerID="b034f0de58aed3530571ccad89bd4b679f11f5076772ab254f53382252757ea8"
Nov 02 13:00:20.269047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.260550    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"]
Nov 02 13:00:20.317768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.317692    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"]
Nov 02 13:00:20.330839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.330807    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893740-cggxb"]
Nov 02 13:00:20.363323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.363284    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"
Nov 02 13:00:20.378448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.378416    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"]
Nov 02 13:00:20.392409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.392372    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893740-44hqc"]
Nov 02 13:00:20.408036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.407920    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7001-111df1"]
Nov 02 13:00:20.412585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.412516    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"]
Nov 02 13:00:20.419404 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.419375    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7001-111df1"]
Nov 02 13:00:20.446025 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.445986    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"]
Nov 02 13:00:20.459755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.459728    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"
Nov 02 13:00:20.459934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.459896    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893740-wxj85"]
Nov 02 13:00:20.493156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.493117    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"]
Nov 02 13:00:20.497877 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.497843    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-1e14d4-29893740-wtgtb"]
Nov 02 13:00:20.514905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.514875    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-kube-api-access-wx9hn\" (UniqueName: \"kubernetes.io/projected/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-kube-api-access-wx9hn\") pod \"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab\" (UID: \"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab\") "
Nov 02 13:00:20.515252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.515235    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-maas-api-service-ca\") pod \"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab\" (UID: \"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab\") "
Nov 02 13:00:20.516125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.516046    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-maas-api-service-ca" pod "67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" (UID: "67d038f7-0b52-4fbd-94eb-5d2ee5e78aab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:20.517387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.517356    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:20.518146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.518124    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-kube-api-access-wx9hn" pod "67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" (UID: "67d038f7-0b52-4fbd-94eb-5d2ee5e78aab"). InnerVolumeSpecName "kube-api-access-wx9hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:20.560187 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.560153    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7001-6ad656"]
Nov 02 13:00:20.561102 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561085    5879 cpu_manager.go:414] "containerMap: removing container" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" containerName="cleanup"
Nov 02 13:00:20.561148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561105    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" containerName="cleanup"
Nov 02 13:00:20.561148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561126    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537" containerName="cleanup"
Nov 02 13:00:20.561148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561134    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537" containerName="cleanup"
Nov 02 13:00:20.561244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561148    5879 cpu_manager.go:414] "containerMap: removing container" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" containerName="cleanup"
Nov 02 13:00:20.561244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561156    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" containerName="cleanup"
Nov 02 13:00:20.561244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561168    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4cf92ef9-e5de-4ba0-a134-661367499778" containerName="cleanup"
Nov 02 13:00:20.561244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561175    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf92ef9-e5de-4ba0-a134-661367499778" containerName="cleanup"
Nov 02 13:00:20.561244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561187    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537" containerName="cleanup"
Nov 02 13:00:20.561244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561194    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537" containerName="cleanup"
Nov 02 13:00:20.561244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561206    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4cf92ef9-e5de-4ba0-a134-661367499778" containerName="cleanup"
Nov 02 13:00:20.561244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561213    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf92ef9-e5de-4ba0-a134-661367499778" containerName="cleanup"
Nov 02 13:00:20.561244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561235    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cc4e5176-3ca6-4719-8555-77b135b17641" containerName="cleanup"
Nov 02 13:00:20.561244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561242    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4e5176-3ca6-4719-8555-77b135b17641" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561255    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cc4e5176-3ca6-4719-8555-77b135b17641" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561263    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4e5176-3ca6-4719-8555-77b135b17641" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561277    5879 cpu_manager.go:414] "containerMap: removing container" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561284    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561303    5879 cpu_manager.go:414] "containerMap: removing container" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561311    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561321    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561328    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561344    5879 cpu_manager.go:414] "containerMap: removing container" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561351    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561394    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561403    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561413    5879 cpu_manager.go:414] "containerMap: removing container" podUID="198a83f1-cd93-42dc-959d-fd4e8db89db5" containerName="test-curl-7001-111df1"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561421    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198a83f1-cd93-42dc-959d-fd4e8db89db5" containerName="test-curl-7001-111df1"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561556    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561568    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561578    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561586    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" containerName="cleanup"
Nov 02 13:00:20.561589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561596    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="198a83f1-cd93-42dc-959d-fd4e8db89db5" containerName="test-curl-7001-111df1"
Nov 02 13:00:20.562184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561608    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4e5176-3ca6-4719-8555-77b135b17641" containerName="cleanup"
Nov 02 13:00:20.562184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561616    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" containerName="cleanup"
Nov 02 13:00:20.562184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561625    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" containerName="cleanup"
Nov 02 13:00:20.562184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561633    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf92ef9-e5de-4ba0-a134-661367499778" containerName="cleanup"
Nov 02 13:00:20.562184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561641    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537" containerName="cleanup"
Nov 02 13:00:20.562184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561649    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf92ef9-e5de-4ba0-a134-661367499778" containerName="cleanup"
Nov 02 13:00:20.562184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561681    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4e5176-3ca6-4719-8555-77b135b17641" containerName="cleanup"
Nov 02 13:00:20.562184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561692    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4e5176-3ca6-4719-8555-77b135b17641" containerName="cleanup"
Nov 02 13:00:20.562184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561702    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" containerName="cleanup"
Nov 02 13:00:20.562184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561712    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf92ef9-e5de-4ba0-a134-661367499778" containerName="cleanup"
Nov 02 13:00:20.562184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.561724    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8" containerName="cleanup"
Nov 02 13:00:20.569228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.569153    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-6ad656"
Nov 02 13:00:20.582968 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.582939    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7001-6ad656"]
Nov 02 13:00:20.603433 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.603406    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"
Nov 02 13:00:20.618082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.618051    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f913d0a-8159-4067-bf4d-293bd88a01e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f913d0a-8159-4067-bf4d-293bd88a01e8-maas-api-service-ca\") pod \"9f913d0a-8159-4067-bf4d-293bd88a01e8\" (UID: \"9f913d0a-8159-4067-bf4d-293bd88a01e8\") "
Nov 02 13:00:20.618291 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.618273    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f913d0a-8159-4067-bf4d-293bd88a01e8-kube-api-access-q4lz6\" (UniqueName: \"kubernetes.io/projected/9f913d0a-8159-4067-bf4d-293bd88a01e8-kube-api-access-q4lz6\") pod \"9f913d0a-8159-4067-bf4d-293bd88a01e8\" (UID: \"9f913d0a-8159-4067-bf4d-293bd88a01e8\") "
Nov 02 13:00:20.618446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.618424    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f913d0a-8159-4067-bf4d-293bd88a01e8-maas-api-service-ca" pod "9f913d0a-8159-4067-bf4d-293bd88a01e8" (UID: "9f913d0a-8159-4067-bf4d-293bd88a01e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:20.618903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.618877    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx9hn\" (UniqueName: \"kubernetes.io/projected/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab-kube-api-access-wx9hn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:20.618903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.618904    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f913d0a-8159-4067-bf4d-293bd88a01e8-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:20.620678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.620643    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f913d0a-8159-4067-bf4d-293bd88a01e8-kube-api-access-q4lz6" pod "9f913d0a-8159-4067-bf4d-293bd88a01e8" (UID: "9f913d0a-8159-4067-bf4d-293bd88a01e8"). InnerVolumeSpecName "kube-api-access-q4lz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:20.719662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.719630    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc3511c-f74f-493c-b446-6f74e4830594-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc3511c-f74f-493c-b446-6f74e4830594-maas-api-service-ca\") pod \"2cc3511c-f74f-493c-b446-6f74e4830594\" (UID: \"2cc3511c-f74f-493c-b446-6f74e4830594\") "
Nov 02 13:00:20.719812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.719705    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc3511c-f74f-493c-b446-6f74e4830594-kube-api-access-ht4pr\" (UniqueName: \"kubernetes.io/projected/2cc3511c-f74f-493c-b446-6f74e4830594-kube-api-access-ht4pr\") pod \"2cc3511c-f74f-493c-b446-6f74e4830594\" (UID: \"2cc3511c-f74f-493c-b446-6f74e4830594\") "
Nov 02 13:00:20.719953 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.719934    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq4t7\" (UniqueName: \"kubernetes.io/projected/15cc2134-4efb-4cfb-8859-313f2e6c9f1f-kube-api-access-qq4t7\") pod \"test-curl-7001-6ad656\" (UID: \"15cc2134-4efb-4cfb-8859-313f2e6c9f1f\") " pod="opendatahub/test-curl-7001-6ad656"
Nov 02 13:00:20.720034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.719987    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc3511c-f74f-493c-b446-6f74e4830594-maas-api-service-ca" pod "2cc3511c-f74f-493c-b446-6f74e4830594" (UID: "2cc3511c-f74f-493c-b446-6f74e4830594"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:20.720255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.720240    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2cc3511c-f74f-493c-b446-6f74e4830594-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:20.720334 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.720257    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4lz6\" (UniqueName: \"kubernetes.io/projected/9f913d0a-8159-4067-bf4d-293bd88a01e8-kube-api-access-q4lz6\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:20.722185 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.722154    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc3511c-f74f-493c-b446-6f74e4830594-kube-api-access-ht4pr" pod "2cc3511c-f74f-493c-b446-6f74e4830594" (UID: "2cc3511c-f74f-493c-b446-6f74e4830594"). InnerVolumeSpecName "kube-api-access-ht4pr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:20.821528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.821440    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq4t7\" (UniqueName: \"kubernetes.io/projected/15cc2134-4efb-4cfb-8859-313f2e6c9f1f-kube-api-access-qq4t7\") pod \"test-curl-7001-6ad656\" (UID: \"15cc2134-4efb-4cfb-8859-313f2e6c9f1f\") " pod="opendatahub/test-curl-7001-6ad656"
Nov 02 13:00:20.821528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.821529    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht4pr\" (UniqueName: \"kubernetes.io/projected/2cc3511c-f74f-493c-b446-6f74e4830594-kube-api-access-ht4pr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:20.846953 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.846921    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq4t7\" (UniqueName: \"kubernetes.io/projected/15cc2134-4efb-4cfb-8859-313f2e6c9f1f-kube-api-access-qq4t7\") pod \"test-curl-7001-6ad656\" (UID: \"15cc2134-4efb-4cfb-8859-313f2e6c9f1f\") " pod="opendatahub/test-curl-7001-6ad656"
Nov 02 13:00:20.850874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.850846    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:20.902483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:20.902450    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-6ad656"
Nov 02 13:00:21.195513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.195485    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"
Nov 02 13:00:21.195947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.195488    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9" event={"ID":"2cc3511c-f74f-493c-b446-6f74e4830594","Type":"ContainerDied","Data":"6b77248c8b1696bf44c3c774c027103a380dc899fc84cabacb06ec8d0c48339d"}
Nov 02 13:00:21.195947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.195646    5879 scope.go:122] "RemoveContainer" containerID="5875bbc363561282deb99c47418102ea538ae32b6d8770afb8e3d003a6a890d0"
Nov 02 13:00:21.207321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.207290    5879 generic.go:356] "Generic (PLEG): container finished" podID="66ad3b6c-5dda-4ae7-9b22-00bc73a388a3" containerID="4b371006abc65a13ee629b0e9e5cee70aa280ec1d886097a51e154233a049350" exitCode=137
Nov 02 13:00:21.209088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.209063    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx" event={"ID":"9f913d0a-8159-4067-bf4d-293bd88a01e8","Type":"ContainerDied","Data":"6a2c3dbf47488d411c17564f8efe913c40d6d62cfac72c33c318bfe00c6ed06b"}
Nov 02 13:00:21.209269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.209072    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"
Nov 02 13:00:21.211745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.211722    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7001-6ad656"]
Nov 02 13:00:21.213750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.213726    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s" event={"ID":"67d038f7-0b52-4fbd-94eb-5d2ee5e78aab","Type":"ContainerDied","Data":"1ca7a38340c08a0d8c49e8c22905d58165b2a33e51ee94f4fba433098472d6e2"}
Nov 02 13:00:21.213854 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.213837    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"
Nov 02 13:00:21.220794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.220742    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7001-111df1" podUID="198a83f1-cd93-42dc-959d-fd4e8db89db5" containerName="test-curl-7001-111df1" containerID="cri-o://905b0e9e8fe04e7f73cf1fc757015fdd1263181d98d888131f2973a2361ee530" gracePeriod=2
Nov 02 13:00:21.224333 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:21.224307    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15cc2134_4efb_4cfb_8859_313f2e6c9f1f.slice/crio-3525bda41bb0950b80e1dca7656730f71766e0a69cc4bfc1b552782000462c5e WatchSource:0}: Error finding container 3525bda41bb0950b80e1dca7656730f71766e0a69cc4bfc1b552782000462c5e: Status 404 returned error can't find the container with id 3525bda41bb0950b80e1dca7656730f71766e0a69cc4bfc1b552782000462c5e
Nov 02 13:00:21.235163 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.235141    5879 scope.go:122] "RemoveContainer" containerID="bb4281aa497b025f6a297bef53dfc0436fad36fa082c952f193d869d7821e914"
Nov 02 13:00:21.253697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.253675    5879 scope.go:122] "RemoveContainer" containerID="afd1ef660ee2feb008b27bb7b65e3d818aa3f8bd4c897423960b44da3765ea13"
Nov 02 13:00:21.260410 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:21.260382    5879 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7001-111df1\" is forbidden: User \"system:node:ip-10-0-134-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-229.ec2.internal' and this object" podUID="198a83f1-cd93-42dc-959d-fd4e8db89db5" pod="opendatahub/test-curl-7001-111df1"
Nov 02 13:00:21.277838 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:21.277796    5879 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7001-111df1\" is forbidden: User \"system:node:ip-10-0-134-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-229.ec2.internal' and this object" podUID="198a83f1-cd93-42dc-959d-fd4e8db89db5" pod="opendatahub/test-curl-7001-111df1"
Nov 02 13:00:21.284278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.284247    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"]
Nov 02 13:00:21.323062 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.323036    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"]
Nov 02 13:00:21.326520 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.326497    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893740-gzqgx"]
Nov 02 13:00:21.361065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.361029    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-1a9648"
Nov 02 13:00:21.374228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.374199    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"]
Nov 02 13:00:21.379284 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.379252    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893740-vwb8s"]
Nov 02 13:00:21.412779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.412744    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"]
Nov 02 13:00:21.419391 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.419323    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893740-bf4x9"]
Nov 02 13:00:21.428501 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:21.428464    5879 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7001-111df1\" is forbidden: User \"system:node:ip-10-0-134-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-229.ec2.internal' and this object" podUID="198a83f1-cd93-42dc-959d-fd4e8db89db5" pod="opendatahub/test-curl-7001-111df1"
Nov 02 13:00:21.440234 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.440208    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"
Nov 02 13:00:21.485296 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:21.485261    5879 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7001-111df1\" is forbidden: User \"system:node:ip-10-0-134-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-229.ec2.internal' and this object" podUID="198a83f1-cd93-42dc-959d-fd4e8db89db5" pod="opendatahub/test-curl-7001-111df1"
Nov 02 13:00:21.531842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.531809    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a653e6-e370-43fe-960e-6810aeeec07c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a653e6-e370-43fe-960e-6810aeeec07c-maas-api-service-ca\") pod \"33a653e6-e370-43fe-960e-6810aeeec07c\" (UID: \"33a653e6-e370-43fe-960e-6810aeeec07c\") "
Nov 02 13:00:21.532052 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.531933    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a653e6-e370-43fe-960e-6810aeeec07c-kube-api-access-spkwm\" (UniqueName: \"kubernetes.io/projected/33a653e6-e370-43fe-960e-6810aeeec07c-kube-api-access-spkwm\") pod \"33a653e6-e370-43fe-960e-6810aeeec07c\" (UID: \"33a653e6-e370-43fe-960e-6810aeeec07c\") "
Nov 02 13:00:21.532152 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.532095    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ad3b6c-5dda-4ae7-9b22-00bc73a388a3-kube-api-access-gxl5g\" (UniqueName: \"kubernetes.io/projected/66ad3b6c-5dda-4ae7-9b22-00bc73a388a3-kube-api-access-gxl5g\") pod \"66ad3b6c-5dda-4ae7-9b22-00bc73a388a3\" (UID: \"66ad3b6c-5dda-4ae7-9b22-00bc73a388a3\") "
Nov 02 13:00:21.532237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.532215    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a653e6-e370-43fe-960e-6810aeeec07c-maas-api-service-ca" pod "33a653e6-e370-43fe-960e-6810aeeec07c" (UID: "33a653e6-e370-43fe-960e-6810aeeec07c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:00:21.532538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.532509    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a653e6-e370-43fe-960e-6810aeeec07c-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:21.534849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.534827    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ad3b6c-5dda-4ae7-9b22-00bc73a388a3-kube-api-access-gxl5g" pod "66ad3b6c-5dda-4ae7-9b22-00bc73a388a3" (UID: "66ad3b6c-5dda-4ae7-9b22-00bc73a388a3"). InnerVolumeSpecName "kube-api-access-gxl5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:21.534951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.534869    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a653e6-e370-43fe-960e-6810aeeec07c-kube-api-access-spkwm" pod "33a653e6-e370-43fe-960e-6810aeeec07c" (UID: "33a653e6-e370-43fe-960e-6810aeeec07c"). InnerVolumeSpecName "kube-api-access-spkwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:21.633566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.633532    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxl5g\" (UniqueName: \"kubernetes.io/projected/66ad3b6c-5dda-4ae7-9b22-00bc73a388a3-kube-api-access-gxl5g\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:21.633566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:21.633561    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spkwm\" (UniqueName: \"kubernetes.io/projected/33a653e6-e370-43fe-960e-6810aeeec07c-kube-api-access-spkwm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:22.030003 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.026911    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc3511c-f74f-493c-b446-6f74e4830594" path="/var/lib/kubelet/pods/2cc3511c-f74f-493c-b446-6f74e4830594/volumes"
Nov 02 13:00:22.031850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.030912    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf92ef9-e5de-4ba0-a134-661367499778" path="/var/lib/kubelet/pods/4cf92ef9-e5de-4ba0-a134-661367499778/volumes"
Nov 02 13:00:22.034761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.034734    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" path="/var/lib/kubelet/pods/51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450/volumes"
Nov 02 13:00:22.035612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.035590    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ad3b6c-5dda-4ae7-9b22-00bc73a388a3" path="/var/lib/kubelet/pods/66ad3b6c-5dda-4ae7-9b22-00bc73a388a3/volumes"
Nov 02 13:00:22.036277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.036259    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" path="/var/lib/kubelet/pods/67d038f7-0b52-4fbd-94eb-5d2ee5e78aab/volumes"
Nov 02 13:00:22.037392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.037373    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8" path="/var/lib/kubelet/pods/9f913d0a-8159-4067-bf4d-293bd88a01e8/volumes"
Nov 02 13:00:22.037981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.037965    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4e5176-3ca6-4719-8555-77b135b17641" path="/var/lib/kubelet/pods/cc4e5176-3ca6-4719-8555-77b135b17641/volumes"
Nov 02 13:00:22.041979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.041958    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537" path="/var/lib/kubelet/pods/ee3505c4-6e81-4604-baf9-b02a5051a537/volumes"
Nov 02 13:00:22.231561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.231532    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-1a9648"
Nov 02 13:00:22.231561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.231555    5879 scope.go:122] "RemoveContainer" containerID="4b371006abc65a13ee629b0e9e5cee70aa280ec1d886097a51e154233a049350"
Nov 02 13:00:22.237422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.237397    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"
Nov 02 13:00:22.237575 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.237435    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk" event={"ID":"33a653e6-e370-43fe-960e-6810aeeec07c","Type":"ContainerDied","Data":"1ba07ef473ab9d361eb7fe72fb98f77b704f93002d86f01307a86dc547a15041"}
Nov 02 13:00:22.242650 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.242618    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7001-6ad656" event={"ID":"15cc2134-4efb-4cfb-8859-313f2e6c9f1f","Type":"ContainerStarted","Data":"ba4ad6bfa2d7c83178c3c2da87c0062f396243a7db71144270f3a08429682eb3"}
Nov 02 13:00:22.242774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.242662    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7001-6ad656" event={"ID":"15cc2134-4efb-4cfb-8859-313f2e6c9f1f","Type":"ContainerStarted","Data":"3525bda41bb0950b80e1dca7656730f71766e0a69cc4bfc1b552782000462c5e"}
Nov 02 13:00:22.245823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.245803    5879 scope.go:122] "RemoveContainer" containerID="b489ed1bea26d11bbfcbf8bd8857b44727b0c983fbbd58e1a484b4c6852f40c5"
Nov 02 13:00:22.299502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.299409    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"]
Nov 02 13:00:22.312077 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.312047    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893740-znmrk"]
Nov 02 13:00:22.355832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.355785    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7001-6ad656" podStartSLOduration=2.35577212 podStartE2EDuration="2.35577212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:22.35321207 +0000 UTC m=+1968.459914924" watchObservedRunningTime="2026-11-02 13:00:22.35577212 +0000 UTC m=+1968.462474972"
Nov 02 13:00:22.587901 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:22.587816    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:23.247104 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.247073    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7001-6ad656"]
Nov 02 13:00:23.256865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.256838    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7001-6ad656"]
Nov 02 13:00:23.395212 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395183    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7001-5b91d4"]
Nov 02 13:00:23.395742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395729    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8" containerName="cleanup"
Nov 02 13:00:23.395808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395743    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8" containerName="cleanup"
Nov 02 13:00:23.395808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395756    5879 cpu_manager.go:414] "containerMap: removing container" podUID="33a653e6-e370-43fe-960e-6810aeeec07c" containerName="cleanup"
Nov 02 13:00:23.395808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395761    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a653e6-e370-43fe-960e-6810aeeec07c" containerName="cleanup"
Nov 02 13:00:23.395808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395784    5879 cpu_manager.go:414] "containerMap: removing container" podUID="33a653e6-e370-43fe-960e-6810aeeec07c" containerName="cleanup"
Nov 02 13:00:23.395808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395789    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a653e6-e370-43fe-960e-6810aeeec07c" containerName="cleanup"
Nov 02 13:00:23.395808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395799    5879 cpu_manager.go:414] "containerMap: removing container" podUID="33a653e6-e370-43fe-960e-6810aeeec07c" containerName="cleanup"
Nov 02 13:00:23.395808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395806    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a653e6-e370-43fe-960e-6810aeeec07c" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395816    5879 cpu_manager.go:414] "containerMap: removing container" podUID="15cc2134-4efb-4cfb-8859-313f2e6c9f1f" containerName="test-curl-7001-6ad656"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395821    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15cc2134-4efb-4cfb-8859-313f2e6c9f1f" containerName="test-curl-7001-6ad656"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395836    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2cc3511c-f74f-493c-b446-6f74e4830594" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395840    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc3511c-f74f-493c-b446-6f74e4830594" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395855    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2cc3511c-f74f-493c-b446-6f74e4830594" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395860    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc3511c-f74f-493c-b446-6f74e4830594" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395869    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2cc3511c-f74f-493c-b446-6f74e4830594" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395873    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc3511c-f74f-493c-b446-6f74e4830594" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395879    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395884    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395891    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4cf92ef9-e5de-4ba0-a134-661367499778" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395895    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf92ef9-e5de-4ba0-a134-661367499778" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395908    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cc4e5176-3ca6-4719-8555-77b135b17641" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395912    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4e5176-3ca6-4719-8555-77b135b17641" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395920    5879 cpu_manager.go:414] "containerMap: removing container" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.395924    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67d038f7-0b52-4fbd-94eb-5d2ee5e78aab" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.396025    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f913d0a-8159-4067-bf4d-293bd88a01e8" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.396035    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a653e6-e370-43fe-960e-6810aeeec07c" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.396045    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a653e6-e370-43fe-960e-6810aeeec07c" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.396051    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a653e6-e370-43fe-960e-6810aeeec07c" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.396055    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="15cc2134-4efb-4cfb-8859-313f2e6c9f1f" containerName="test-curl-7001-6ad656"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.396063    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc3511c-f74f-493c-b446-6f74e4830594" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.396069    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc3511c-f74f-493c-b446-6f74e4830594" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.396075    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="51f8e66d-8dd1-413c-a1c9-0fc2dbb0a450" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.396082    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc3511c-f74f-493c-b446-6f74e4830594" containerName="cleanup"
Nov 02 13:00:23.396081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.396088    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3505c4-6e81-4604-baf9-b02a5051a537" containerName="cleanup"
Nov 02 13:00:23.400663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.400642    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-5b91d4"
Nov 02 13:00:23.419506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.418634    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7001-5b91d4"]
Nov 02 13:00:23.472408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.472378    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-111df1"
Nov 02 13:00:23.559050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.558952    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198a83f1-cd93-42dc-959d-fd4e8db89db5-kube-api-access-87xr6\" (UniqueName: \"kubernetes.io/projected/198a83f1-cd93-42dc-959d-fd4e8db89db5-kube-api-access-87xr6\") pod \"198a83f1-cd93-42dc-959d-fd4e8db89db5\" (UID: \"198a83f1-cd93-42dc-959d-fd4e8db89db5\") "
Nov 02 13:00:23.559220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.559169    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwqx\" (UniqueName: \"kubernetes.io/projected/ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982-kube-api-access-fpwqx\") pod \"test-curl-7001-5b91d4\" (UID: \"ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982\") " pod="opendatahub/test-curl-7001-5b91d4"
Nov 02 13:00:23.561216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.561191    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198a83f1-cd93-42dc-959d-fd4e8db89db5-kube-api-access-87xr6" pod "198a83f1-cd93-42dc-959d-fd4e8db89db5" (UID: "198a83f1-cd93-42dc-959d-fd4e8db89db5"). InnerVolumeSpecName "kube-api-access-87xr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:23.660780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.660746    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwqx\" (UniqueName: \"kubernetes.io/projected/ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982-kube-api-access-fpwqx\") pod \"test-curl-7001-5b91d4\" (UID: \"ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982\") " pod="opendatahub/test-curl-7001-5b91d4"
Nov 02 13:00:23.661096 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.661076    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87xr6\" (UniqueName: \"kubernetes.io/projected/198a83f1-cd93-42dc-959d-fd4e8db89db5-kube-api-access-87xr6\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:23.686002 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.685968    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwqx\" (UniqueName: \"kubernetes.io/projected/ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982-kube-api-access-fpwqx\") pod \"test-curl-7001-5b91d4\" (UID: \"ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982\") " pod="opendatahub/test-curl-7001-5b91d4"
Nov 02 13:00:23.724497 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:23.724463    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-5b91d4"
Nov 02 13:00:24.017964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.017930    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198a83f1-cd93-42dc-959d-fd4e8db89db5" path="/var/lib/kubelet/pods/198a83f1-cd93-42dc-959d-fd4e8db89db5/volumes"
Nov 02 13:00:24.018491 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.018470    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a653e6-e370-43fe-960e-6810aeeec07c" path="/var/lib/kubelet/pods/33a653e6-e370-43fe-960e-6810aeeec07c/volumes"
Nov 02 13:00:24.035286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.034480    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7001-5b91d4"]
Nov 02 13:00:24.263753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.263714    5879 generic.go:356] "Generic (PLEG): container finished" podID="198a83f1-cd93-42dc-959d-fd4e8db89db5" containerID="905b0e9e8fe04e7f73cf1fc757015fdd1263181d98d888131f2973a2361ee530" exitCode=137
Nov 02 13:00:24.264270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.263793    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-111df1"
Nov 02 13:00:24.264270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.263827    5879 scope.go:122] "RemoveContainer" containerID="905b0e9e8fe04e7f73cf1fc757015fdd1263181d98d888131f2973a2361ee530"
Nov 02 13:00:24.266452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.266420    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7001-6ad656" podUID="15cc2134-4efb-4cfb-8859-313f2e6c9f1f" containerName="test-curl-7001-6ad656" containerID="cri-o://ba4ad6bfa2d7c83178c3c2da87c0062f396243a7db71144270f3a08429682eb3" gracePeriod=2
Nov 02 13:00:24.267006 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.266962    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7001-5b91d4" event={"ID":"ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982","Type":"ContainerStarted","Data":"295150457ffea44795cc8823fb576c712881ea72aa9d992fc57bb5cf09212e60"}
Nov 02 13:00:24.267439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.267053    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7001-5b91d4" event={"ID":"ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982","Type":"ContainerStarted","Data":"3c4e7465dac871c3b37afc411fbc627626b5e3c0194ef5bc43a38ddc1d911359"}
Nov 02 13:00:24.276927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.276903    5879 scope.go:122] "RemoveContainer" containerID="905b0e9e8fe04e7f73cf1fc757015fdd1263181d98d888131f2973a2361ee530"
Nov 02 13:00:24.277633 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:24.277602    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905b0e9e8fe04e7f73cf1fc757015fdd1263181d98d888131f2973a2361ee530\": container with ID starting with 905b0e9e8fe04e7f73cf1fc757015fdd1263181d98d888131f2973a2361ee530 not found: ID does not exist" containerID="905b0e9e8fe04e7f73cf1fc757015fdd1263181d98d888131f2973a2361ee530"
Nov 02 13:00:24.277804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.277649    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905b0e9e8fe04e7f73cf1fc757015fdd1263181d98d888131f2973a2361ee530"} err="failed to get container status \"905b0e9e8fe04e7f73cf1fc757015fdd1263181d98d888131f2973a2361ee530\": rpc error: code = NotFound desc = could not find container \"905b0e9e8fe04e7f73cf1fc757015fdd1263181d98d888131f2973a2361ee530\": container with ID starting with 905b0e9e8fe04e7f73cf1fc757015fdd1263181d98d888131f2973a2361ee530 not found: ID does not exist"
Nov 02 13:00:24.314257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.314194    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7001-5b91d4" podStartSLOduration=1.314175024 podStartE2EDuration="1.314175024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:24.311538197 +0000 UTC m=+1970.418241051" watchObservedRunningTime="2026-11-02 13:00:24.314175024 +0000 UTC m=+1970.420877877"
Nov 02 13:00:24.355106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:24.355072    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:25.177794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:25.177756    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7001-5b91d4"]
Nov 02 13:00:25.184957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:25.184925    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7001-5b91d4"]
Nov 02 13:00:25.876599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:25.876564    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:26.165760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:26.165659    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:26.283641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:26.283591    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7001-5b91d4" podUID="ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982" containerName="test-curl-7001-5b91d4" containerID="cri-o://295150457ffea44795cc8823fb576c712881ea72aa9d992fc57bb5cf09212e60" gracePeriod=2
Nov 02 13:00:26.498764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:26.498734    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-6ad656"
Nov 02 13:00:26.598749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:26.598717    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15cc2134-4efb-4cfb-8859-313f2e6c9f1f-kube-api-access-qq4t7\" (UniqueName: \"kubernetes.io/projected/15cc2134-4efb-4cfb-8859-313f2e6c9f1f-kube-api-access-qq4t7\") pod \"15cc2134-4efb-4cfb-8859-313f2e6c9f1f\" (UID: \"15cc2134-4efb-4cfb-8859-313f2e6c9f1f\") "
Nov 02 13:00:26.600957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:26.600931    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15cc2134-4efb-4cfb-8859-313f2e6c9f1f-kube-api-access-qq4t7" pod "15cc2134-4efb-4cfb-8859-313f2e6c9f1f" (UID: "15cc2134-4efb-4cfb-8859-313f2e6c9f1f"). InnerVolumeSpecName "kube-api-access-qq4t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:26.700957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:26.700923    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq4t7\" (UniqueName: \"kubernetes.io/projected/15cc2134-4efb-4cfb-8859-313f2e6c9f1f-kube-api-access-qq4t7\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:26.725984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:26.725891    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:27.139518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:27.139482    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:27.269019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:27.268580    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:27.313021 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:27.312403    5879 generic.go:356] "Generic (PLEG): container finished" podID="15cc2134-4efb-4cfb-8859-313f2e6c9f1f" containerID="ba4ad6bfa2d7c83178c3c2da87c0062f396243a7db71144270f3a08429682eb3" exitCode=137
Nov 02 13:00:27.313021 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:27.312457    5879 scope.go:122] "RemoveContainer" containerID="ba4ad6bfa2d7c83178c3c2da87c0062f396243a7db71144270f3a08429682eb3"
Nov 02 13:00:27.313021 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:27.312552    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-6ad656"
Nov 02 13:00:27.337026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:27.336971    5879 scope.go:122] "RemoveContainer" containerID="ba4ad6bfa2d7c83178c3c2da87c0062f396243a7db71144270f3a08429682eb3"
Nov 02 13:00:27.337490 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:27.337449    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4ad6bfa2d7c83178c3c2da87c0062f396243a7db71144270f3a08429682eb3\": container with ID starting with ba4ad6bfa2d7c83178c3c2da87c0062f396243a7db71144270f3a08429682eb3 not found: ID does not exist" containerID="ba4ad6bfa2d7c83178c3c2da87c0062f396243a7db71144270f3a08429682eb3"
Nov 02 13:00:27.337583 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:27.337497    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4ad6bfa2d7c83178c3c2da87c0062f396243a7db71144270f3a08429682eb3"} err="failed to get container status \"ba4ad6bfa2d7c83178c3c2da87c0062f396243a7db71144270f3a08429682eb3\": rpc error: code = NotFound desc = could not find container \"ba4ad6bfa2d7c83178c3c2da87c0062f396243a7db71144270f3a08429682eb3\": container with ID starting with ba4ad6bfa2d7c83178c3c2da87c0062f396243a7db71144270f3a08429682eb3 not found: ID does not exist"
Nov 02 13:00:27.345168 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:27.345130    5879 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7001-6ad656\" is forbidden: User \"system:node:ip-10-0-134-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-229.ec2.internal' and this object" podUID="15cc2134-4efb-4cfb-8859-313f2e6c9f1f" pod="opendatahub/test-curl-7001-6ad656"
Nov 02 13:00:27.481703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:27.481621    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:28.018567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:28.018488    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15cc2134-4efb-4cfb-8859-313f2e6c9f1f" path="/var/lib/kubelet/pods/15cc2134-4efb-4cfb-8859-313f2e6c9f1f/volumes"
Nov 02 13:00:28.491734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:28.491703    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-5b91d4"
Nov 02 13:00:28.629907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:28.629872    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982-kube-api-access-fpwqx\" (UniqueName: \"kubernetes.io/projected/ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982-kube-api-access-fpwqx\") pod \"ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982\" (UID: \"ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982\") "
Nov 02 13:00:28.632352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:28.632319    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982-kube-api-access-fpwqx" pod "ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982" (UID: "ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982"). InnerVolumeSpecName "kube-api-access-fpwqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:28.731531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:28.731489    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpwqx\" (UniqueName: \"kubernetes.io/projected/ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982-kube-api-access-fpwqx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:29.129816 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:29.129769    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:29.160613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:29.160582    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"
Nov 02 13:00:29.283903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:29.283869    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:29.330737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:29.330700    5879 generic.go:356] "Generic (PLEG): container finished" podID="ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982" containerID="295150457ffea44795cc8823fb576c712881ea72aa9d992fc57bb5cf09212e60" exitCode=137
Nov 02 13:00:29.330943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:29.330748    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7001-5b91d4"
Nov 02 13:00:29.330943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:29.330803    5879 scope.go:122] "RemoveContainer" containerID="295150457ffea44795cc8823fb576c712881ea72aa9d992fc57bb5cf09212e60"
Nov 02 13:00:29.344393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:29.344368    5879 scope.go:122] "RemoveContainer" containerID="295150457ffea44795cc8823fb576c712881ea72aa9d992fc57bb5cf09212e60"
Nov 02 13:00:29.344766 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:29.344743    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"295150457ffea44795cc8823fb576c712881ea72aa9d992fc57bb5cf09212e60\": container with ID starting with 295150457ffea44795cc8823fb576c712881ea72aa9d992fc57bb5cf09212e60 not found: ID does not exist" containerID="295150457ffea44795cc8823fb576c712881ea72aa9d992fc57bb5cf09212e60"
Nov 02 13:00:29.344865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:29.344773    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295150457ffea44795cc8823fb576c712881ea72aa9d992fc57bb5cf09212e60"} err="failed to get container status \"295150457ffea44795cc8823fb576c712881ea72aa9d992fc57bb5cf09212e60\": rpc error: code = NotFound desc = could not find container \"295150457ffea44795cc8823fb576c712881ea72aa9d992fc57bb5cf09212e60\": container with ID starting with 295150457ffea44795cc8823fb576c712881ea72aa9d992fc57bb5cf09212e60 not found: ID does not exist"
Nov 02 13:00:29.357530 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:29.357500    5879 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7001-5b91d4\" is forbidden: User \"system:node:ip-10-0-134-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-229.ec2.internal' and this object" podUID="ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982" pod="opendatahub/test-curl-7001-5b91d4"
Nov 02 13:00:30.017839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:30.017805    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982" path="/var/lib/kubelet/pods/ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982/volumes"
Nov 02 13:00:30.998428 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:30.998383    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:31.136340 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:31.136308    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"]
Nov 02 13:00:31.139154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:31.139116    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:31.278743 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:31.278545    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:32.167847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:32.167810    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"]
Nov 02 13:00:32.199136 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:32.199104    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:32.741917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:32.741880    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:34.021370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.021338    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:34.167042 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.167004    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"]
Nov 02 13:00:34.167641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.167621    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982" containerName="test-curl-7001-5b91d4"
Nov 02 13:00:34.167746 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.167643    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982" containerName="test-curl-7001-5b91d4"
Nov 02 13:00:34.167848 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.167834    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2dfaeb-1d4f-42d9-83b8-af5f9b04e982" containerName="test-curl-7001-5b91d4"
Nov 02 13:00:34.172655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.172635    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"
Nov 02 13:00:34.194620 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.194576    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:34.195301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.195277    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"]
Nov 02 13:00:34.204540 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.204516    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-035dac\""
Nov 02 13:00:34.204540 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.204531    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-035dac-dockercfg-z89hr\""
Nov 02 13:00:34.209640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.209610    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqd8r\" (UniqueName: \"kubernetes.io/projected/14e890f7-843c-478e-97d5-4f343aaa9ad4-kube-api-access-vqd8r\") pod \"payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm\" (UID: \"14e890f7-843c-478e-97d5-4f343aaa9ad4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"
Nov 02 13:00:34.209774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.209666    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e890f7-843c-478e-97d5-4f343aaa9ad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm\" (UID: \"14e890f7-843c-478e-97d5-4f343aaa9ad4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"
Nov 02 13:00:34.316569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.316471    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqd8r\" (UniqueName: \"kubernetes.io/projected/14e890f7-843c-478e-97d5-4f343aaa9ad4-kube-api-access-vqd8r\") pod \"payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm\" (UID: \"14e890f7-843c-478e-97d5-4f343aaa9ad4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"
Nov 02 13:00:34.317204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.317172    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e890f7-843c-478e-97d5-4f343aaa9ad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm\" (UID: \"14e890f7-843c-478e-97d5-4f343aaa9ad4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"
Nov 02 13:00:34.318337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.318158    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e890f7-843c-478e-97d5-4f343aaa9ad4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm\" (UID: \"14e890f7-843c-478e-97d5-4f343aaa9ad4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"
Nov 02 13:00:34.319794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.319772    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"]
Nov 02 13:00:34.325905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.325886    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"
Nov 02 13:00:34.373476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.373433    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"]
Nov 02 13:00:34.387231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.387200    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqd8r\" (UniqueName: \"kubernetes.io/projected/14e890f7-843c-478e-97d5-4f343aaa9ad4-kube-api-access-vqd8r\") pod \"payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm\" (UID: \"14e890f7-843c-478e-97d5-4f343aaa9ad4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"
Nov 02 13:00:34.419455 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.419416    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj4bw\" (UniqueName: \"kubernetes.io/projected/97ae8f04-3aba-4de2-9987-4f4bc895e558-kube-api-access-bj4bw\") pod \"payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg\" (UID: \"97ae8f04-3aba-4de2-9987-4f4bc895e558\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"
Nov 02 13:00:34.419628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.419474    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ae8f04-3aba-4de2-9987-4f4bc895e558-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg\" (UID: \"97ae8f04-3aba-4de2-9987-4f4bc895e558\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"
Nov 02 13:00:34.487523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.487490    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"
Nov 02 13:00:34.520726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.520687    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj4bw\" (UniqueName: \"kubernetes.io/projected/97ae8f04-3aba-4de2-9987-4f4bc895e558-kube-api-access-bj4bw\") pod \"payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg\" (UID: \"97ae8f04-3aba-4de2-9987-4f4bc895e558\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"
Nov 02 13:00:34.521674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.520753    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ae8f04-3aba-4de2-9987-4f4bc895e558-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg\" (UID: \"97ae8f04-3aba-4de2-9987-4f4bc895e558\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"
Nov 02 13:00:34.521674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.521550    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ae8f04-3aba-4de2-9987-4f4bc895e558-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg\" (UID: \"97ae8f04-3aba-4de2-9987-4f4bc895e558\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"
Nov 02 13:00:34.548174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.548099    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:34.553551 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.553523    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj4bw\" (UniqueName: \"kubernetes.io/projected/97ae8f04-3aba-4de2-9987-4f4bc895e558-kube-api-access-bj4bw\") pod \"payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg\" (UID: \"97ae8f04-3aba-4de2-9987-4f4bc895e558\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"
Nov 02 13:00:34.643833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.643795    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"
Nov 02 13:00:34.818194 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:34.818144    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e890f7_843c_478e_97d5_4f343aaa9ad4.slice/crio-2381c4216d93d9987a87bef4967c5af4915da392b93b6586a7084767ae42acb5 WatchSource:0}: Error finding container 2381c4216d93d9987a87bef4967c5af4915da392b93b6586a7084767ae42acb5: Status 404 returned error can't find the container with id 2381c4216d93d9987a87bef4967c5af4915da392b93b6586a7084767ae42acb5
Nov 02 13:00:34.818599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.818566    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"]
Nov 02 13:00:34.951138 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:34.951111    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ae8f04_3aba_4de2_9987_4f4bc895e558.slice/crio-40ff928dbf4322f159a05134b98de0315027f4292b2635fc66a78b5372709433 WatchSource:0}: Error finding container 40ff928dbf4322f159a05134b98de0315027f4292b2635fc66a78b5372709433: Status 404 returned error can't find the container with id 40ff928dbf4322f159a05134b98de0315027f4292b2635fc66a78b5372709433
Nov 02 13:00:34.951861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:34.951840    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"]
Nov 02 13:00:35.385588 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:35.385534    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg" event={"ID":"97ae8f04-3aba-4de2-9987-4f4bc895e558","Type":"ContainerStarted","Data":"72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1"}
Nov 02 13:00:35.386087 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:35.385606    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg" event={"ID":"97ae8f04-3aba-4de2-9987-4f4bc895e558","Type":"ContainerStarted","Data":"40ff928dbf4322f159a05134b98de0315027f4292b2635fc66a78b5372709433"}
Nov 02 13:00:35.386087 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:35.385716    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"
Nov 02 13:00:35.387886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:35.387704    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm" event={"ID":"14e890f7-843c-478e-97d5-4f343aaa9ad4","Type":"ContainerStarted","Data":"44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a"}
Nov 02 13:00:35.387886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:35.387750    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm" event={"ID":"14e890f7-843c-478e-97d5-4f343aaa9ad4","Type":"ContainerStarted","Data":"2381c4216d93d9987a87bef4967c5af4915da392b93b6586a7084767ae42acb5"}
Nov 02 13:00:35.388341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:35.388293    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"
Nov 02 13:00:35.431757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:35.431703    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg" podStartSLOduration=1.4316861969999999 podStartE2EDuration="1.431686197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:35.429972734 +0000 UTC m=+1981.536675586" watchObservedRunningTime="2026-11-02 13:00:35.431686197 +0000 UTC m=+1981.538389050"
Nov 02 13:00:35.477114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:35.477038    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm" podStartSLOduration=1.4770187 podStartE2EDuration="1.4770187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:35.473729223 +0000 UTC m=+1981.580432074" watchObservedRunningTime="2026-11-02 13:00:35.4770187 +0000 UTC m=+1981.583721544"
Nov 02 13:00:35.783386 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:35.783297    5879 scope.go:122] "RemoveContainer" containerID="673e0159cf29ef458c209aedd01897298a6cfa0f0986d64c7ee257f83577016c"
Nov 02 13:00:35.794916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:35.794879    5879 scope.go:122] "RemoveContainer" containerID="fa45a2ff95c0734f8d30ab509f4d7044b90beef66552864d77fa59c9e785991d"
Nov 02 13:00:36.071980 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:36.071896    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:37.181959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:37.181929    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:37.638098 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:37.638065    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:39.247085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:39.247057    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:40.808024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:40.807962    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:40.933638 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:40.933603    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:41.141518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:41.141479    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:41.441078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:41.436499    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:42.649863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:42.649826    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:42.991092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:42.990949    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:43.598331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:43.598246    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:43.724016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:43.719403    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:44.393965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:44.393930    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:44.821652 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:44.821576    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:00:45.867010 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:45.866969    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:46.160528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.160425    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz" podUID="f5b47ed8-a443-4f92-9589-3288fb61d71d" containerName="main" containerID="cri-o://cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c" gracePeriod=45
Nov 02 13:00:46.399102 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.399074    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"
Nov 02 13:00:46.400381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.400167    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"
Nov 02 13:00:46.472937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.472903    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 13:00:46.477485 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.477456    5879 generic.go:356] "Generic (PLEG): container finished" podID="f5b47ed8-a443-4f92-9589-3288fb61d71d" containerID="cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c" exitCode=0
Nov 02 13:00:46.477626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.477503    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"
Nov 02 13:00:46.477626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.477542    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz" event={"ID":"f5b47ed8-a443-4f92-9589-3288fb61d71d","Type":"ContainerDied","Data":"cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c"}
Nov 02 13:00:46.477626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.477594    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz" event={"ID":"f5b47ed8-a443-4f92-9589-3288fb61d71d","Type":"ContainerDied","Data":"b69870e1be0d3ce3468d5da982acdc7a4abbdda5141fc67b073e3917abd371ae"}
Nov 02 13:00:46.477626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.477619    5879 scope.go:122] "RemoveContainer" containerID="cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c"
Nov 02 13:00:46.490523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.490496    5879 scope.go:122] "RemoveContainer" containerID="cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c"
Nov 02 13:00:46.490919 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:46.490891    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c\": container with ID starting with cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c not found: ID does not exist" containerID="cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c"
Nov 02 13:00:46.491116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.490927    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c"} err="failed to get container status \"cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c\": rpc error: code = NotFound desc = could not find container \"cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c\": container with ID starting with cbada0552aecde80ceaacbe817ece09d8d46ffdc8e85140c3ff7c7e148caf22c not found: ID does not exist"
Nov 02 13:00:46.573696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.573653    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:46.587518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.587473    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b47ed8-a443-4f92-9589-3288fb61d71d-kube-api-access-zc8wn\" (UniqueName: \"kubernetes.io/projected/f5b47ed8-a443-4f92-9589-3288fb61d71d-kube-api-access-zc8wn\") pod \"f5b47ed8-a443-4f92-9589-3288fb61d71d\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") "
Nov 02 13:00:46.587717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.587602    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5b47ed8-a443-4f92-9589-3288fb61d71d-tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b47ed8-a443-4f92-9589-3288fb61d71d-tls-certs\") pod \"f5b47ed8-a443-4f92-9589-3288fb61d71d\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") "
Nov 02 13:00:46.587717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.587670    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-dshm\") pod \"f5b47ed8-a443-4f92-9589-3288fb61d71d\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") "
Nov 02 13:00:46.587849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.587733    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-home\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-home\") pod \"f5b47ed8-a443-4f92-9589-3288fb61d71d\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") "
Nov 02 13:00:46.587910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.587853    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-model-cache\") pod \"f5b47ed8-a443-4f92-9589-3288fb61d71d\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") "
Nov 02 13:00:46.587978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.587910    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-tmp-dir\") pod \"f5b47ed8-a443-4f92-9589-3288fb61d71d\" (UID: \"f5b47ed8-a443-4f92-9589-3288fb61d71d\") "
Nov 02 13:00:46.588842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.588792    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-tmp-dir" pod "f5b47ed8-a443-4f92-9589-3288fb61d71d" (UID: "f5b47ed8-a443-4f92-9589-3288fb61d71d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:46.588842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.588872    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-model-cache" pod "f5b47ed8-a443-4f92-9589-3288fb61d71d" (UID: "f5b47ed8-a443-4f92-9589-3288fb61d71d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:46.590890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.590864    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-dshm" pod "f5b47ed8-a443-4f92-9589-3288fb61d71d" (UID: "f5b47ed8-a443-4f92-9589-3288fb61d71d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:46.591194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.591173    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-home" pod "f5b47ed8-a443-4f92-9589-3288fb61d71d" (UID: "f5b47ed8-a443-4f92-9589-3288fb61d71d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:46.591607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.591579    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b47ed8-a443-4f92-9589-3288fb61d71d-kube-api-access-zc8wn" pod "f5b47ed8-a443-4f92-9589-3288fb61d71d" (UID: "f5b47ed8-a443-4f92-9589-3288fb61d71d"). InnerVolumeSpecName "kube-api-access-zc8wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:46.593181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.593159    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5b47ed8-a443-4f92-9589-3288fb61d71d-tls-certs" pod "f5b47ed8-a443-4f92-9589-3288fb61d71d" (UID: "f5b47ed8-a443-4f92-9589-3288fb61d71d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:46.690235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.690157    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5b47ed8-a443-4f92-9589-3288fb61d71d-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:46.690235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.690187    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:46.690235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.690195    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:46.690235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.690203    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:46.690235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.690212    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5b47ed8-a443-4f92-9589-3288fb61d71d-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:46.690235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.690221    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8wn\" (UniqueName: \"kubernetes.io/projected/f5b47ed8-a443-4f92-9589-3288fb61d71d-kube-api-access-zc8wn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:46.829421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.829388    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"]
Nov 02 13:00:46.842339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.842310    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz"]
Nov 02 13:00:46.942211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:46.942141    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:47.084468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.084436    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:47.190327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.190289    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t" podUID="1176222b-0fa2-40d2-898b-9a6fcd8d35aa" containerName="main" containerID="cri-o://6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3" gracePeriod=45
Nov 02 13:00:47.411116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.411060    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/auth-test-model-a6f2c0-kserve-7d9864f797-87dzz" podUID="f5b47ed8-a443-4f92-9589-3288fb61d71d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.164:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:00:47.477505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.477477    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 13:00:47.486245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.486209    5879 generic.go:356] "Generic (PLEG): container finished" podID="1176222b-0fa2-40d2-898b-9a6fcd8d35aa" containerID="6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3" exitCode=0
Nov 02 13:00:47.486427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.486255    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"
Nov 02 13:00:47.486427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.486307    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t" event={"ID":"1176222b-0fa2-40d2-898b-9a6fcd8d35aa","Type":"ContainerDied","Data":"6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3"}
Nov 02 13:00:47.486427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.486360    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t" event={"ID":"1176222b-0fa2-40d2-898b-9a6fcd8d35aa","Type":"ContainerDied","Data":"d9fd3548ac861667a66023c5d3e0398fc165da3b1f9e1e12e492a606b98e43ca"}
Nov 02 13:00:47.486427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.486382    5879 scope.go:122] "RemoveContainer" containerID="6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3"
Nov 02 13:00:47.501059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.501005    5879 scope.go:122] "RemoveContainer" containerID="6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3"
Nov 02 13:00:47.501356 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:47.501328    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3\": container with ID starting with 6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3 not found: ID does not exist" containerID="6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3"
Nov 02 13:00:47.501455 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.501366    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3"} err="failed to get container status \"6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3\": rpc error: code = NotFound desc = could not find container \"6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3\": container with ID starting with 6716807f37bdfad1dd60eadc5551fdff07e37fe88948e5151687aba5eeb5a5f3 not found: ID does not exist"
Nov 02 13:00:47.600900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.600865    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tmp-dir\") pod \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") "
Nov 02 13:00:47.601106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.600958    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-kube-api-access-pws6m\" (UniqueName: \"kubernetes.io/projected/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-kube-api-access-pws6m\") pod \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") "
Nov 02 13:00:47.601106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.601079    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-home\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-home\") pod \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") "
Nov 02 13:00:47.601235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.601147    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tmp-dir" pod "1176222b-0fa2-40d2-898b-9a6fcd8d35aa" (UID: "1176222b-0fa2-40d2-898b-9a6fcd8d35aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:47.601235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.601181    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tls-certs\") pod \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") "
Nov 02 13:00:47.601365 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.601249    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-dshm\") pod \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") "
Nov 02 13:00:47.601365 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.601285    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-model-cache\") pod \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\" (UID: \"1176222b-0fa2-40d2-898b-9a6fcd8d35aa\") "
Nov 02 13:00:47.601365 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.601287    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-home" pod "1176222b-0fa2-40d2-898b-9a6fcd8d35aa" (UID: "1176222b-0fa2-40d2-898b-9a6fcd8d35aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:47.601523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.601508    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-model-cache" pod "1176222b-0fa2-40d2-898b-9a6fcd8d35aa" (UID: "1176222b-0fa2-40d2-898b-9a6fcd8d35aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:47.601861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.601840    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:47.601861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.601860    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:47.602320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.601872    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:47.604174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.604152    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-dshm" pod "1176222b-0fa2-40d2-898b-9a6fcd8d35aa" (UID: "1176222b-0fa2-40d2-898b-9a6fcd8d35aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:00:47.604174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.604162    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-kube-api-access-pws6m" pod "1176222b-0fa2-40d2-898b-9a6fcd8d35aa" (UID: "1176222b-0fa2-40d2-898b-9a6fcd8d35aa"). InnerVolumeSpecName "kube-api-access-pws6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:47.604360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.604237    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tls-certs" pod "1176222b-0fa2-40d2-898b-9a6fcd8d35aa" (UID: "1176222b-0fa2-40d2-898b-9a6fcd8d35aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:00:47.703493 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.703410    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pws6m\" (UniqueName: \"kubernetes.io/projected/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-kube-api-access-pws6m\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:47.703493 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.703444    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:47.703493 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.703457    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1176222b-0fa2-40d2-898b-9a6fcd8d35aa-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:47.836004 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.835973    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"]
Nov 02 13:00:47.849227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:47.849197    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t"]
Nov 02 13:00:48.017383 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.017304    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1176222b-0fa2-40d2-898b-9a6fcd8d35aa" path="/var/lib/kubelet/pods/1176222b-0fa2-40d2-898b-9a6fcd8d35aa/volumes"
Nov 02 13:00:48.017771 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.017754    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b47ed8-a443-4f92-9589-3288fb61d71d" path="/var/lib/kubelet/pods/f5b47ed8-a443-4f92-9589-3288fb61d71d/volumes"
Nov 02 13:00:48.435214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.435173    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-ed56b9/auth-test-model-ed56b9-kserve-7cb678bd65-5hv4t" podUID="1176222b-0fa2-40d2-898b-9a6fcd8d35aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.165:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:00:48.524222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.524183    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:48.690276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.690175    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:48.810770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.810743    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:48.850159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.850126    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"]
Nov 02 13:00:48.850765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.850751    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1176222b-0fa2-40d2-898b-9a6fcd8d35aa" containerName="main"
Nov 02 13:00:48.850811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.850767    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1176222b-0fa2-40d2-898b-9a6fcd8d35aa" containerName="main"
Nov 02 13:00:48.850811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.850780    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f5b47ed8-a443-4f92-9589-3288fb61d71d" containerName="main"
Nov 02 13:00:48.850811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.850786    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b47ed8-a443-4f92-9589-3288fb61d71d" containerName="main"
Nov 02 13:00:48.850929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.850919    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1176222b-0fa2-40d2-898b-9a6fcd8d35aa" containerName="main"
Nov 02 13:00:48.850963 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.850933    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b47ed8-a443-4f92-9589-3288fb61d71d" containerName="main"
Nov 02 13:00:48.855970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.855950    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"
Nov 02 13:00:48.886478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.886408    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-25f7a6-dockercfg-vb4sd\""
Nov 02 13:00:48.886478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.886408    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-25f7a6\""
Nov 02 13:00:48.890884 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.890852    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"]
Nov 02 13:00:48.963527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:48.963424    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"]
Nov 02 13:00:49.029980 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.027814    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"]
Nov 02 13:00:49.029980 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.027966    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"
Nov 02 13:00:49.032412 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.032373    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fzcr\" (UniqueName: \"kubernetes.io/projected/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-kube-api-access-2fzcr\") pod \"payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl\" (UID: \"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"
Nov 02 13:00:49.032591 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.032557    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl\" (UID: \"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"
Nov 02 13:00:49.134061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.133982    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f133de5-ca30-497b-a5b3-8eb19ba650e1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z\" (UID: \"5f133de5-ca30-497b-a5b3-8eb19ba650e1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"
Nov 02 13:00:49.134263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.134098    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fzcr\" (UniqueName: \"kubernetes.io/projected/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-kube-api-access-2fzcr\") pod \"payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl\" (UID: \"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"
Nov 02 13:00:49.134336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.134269    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl\" (UID: \"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"
Nov 02 13:00:49.134336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.134303    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2gjl\" (UniqueName: \"kubernetes.io/projected/5f133de5-ca30-497b-a5b3-8eb19ba650e1-kube-api-access-f2gjl\") pod \"payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z\" (UID: \"5f133de5-ca30-497b-a5b3-8eb19ba650e1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"
Nov 02 13:00:49.134958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.134935    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl\" (UID: \"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"
Nov 02 13:00:49.159215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.159182    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fzcr\" (UniqueName: \"kubernetes.io/projected/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-kube-api-access-2fzcr\") pod \"payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl\" (UID: \"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"
Nov 02 13:00:49.170479 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.170434    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"
Nov 02 13:00:49.244409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.241955    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2gjl\" (UniqueName: \"kubernetes.io/projected/5f133de5-ca30-497b-a5b3-8eb19ba650e1-kube-api-access-f2gjl\") pod \"payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z\" (UID: \"5f133de5-ca30-497b-a5b3-8eb19ba650e1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"
Nov 02 13:00:49.244409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.242160    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f133de5-ca30-497b-a5b3-8eb19ba650e1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z\" (UID: \"5f133de5-ca30-497b-a5b3-8eb19ba650e1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"
Nov 02 13:00:49.244409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.243150    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f133de5-ca30-497b-a5b3-8eb19ba650e1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z\" (UID: \"5f133de5-ca30-497b-a5b3-8eb19ba650e1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"
Nov 02 13:00:49.267639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.267604    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2gjl\" (UniqueName: \"kubernetes.io/projected/5f133de5-ca30-497b-a5b3-8eb19ba650e1-kube-api-access-f2gjl\") pod \"payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z\" (UID: \"5f133de5-ca30-497b-a5b3-8eb19ba650e1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"
Nov 02 13:00:49.354734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.354690    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"
Nov 02 13:00:49.417025 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.415796    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"]
Nov 02 13:00:49.485684 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.485568    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"]
Nov 02 13:00:49.486668 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.486641    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:49.490594 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:49.490564    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3a98fb2_a000_4e7a_ae1e_26d3cea3cdfe.slice/crio-576ed9ff7d101f3e08fed79a8a4d9587b3fd62fed2cc67734ec1f519330fefaf WatchSource:0}: Error finding container 576ed9ff7d101f3e08fed79a8a4d9587b3fd62fed2cc67734ec1f519330fefaf: Status 404 returned error can't find the container with id 576ed9ff7d101f3e08fed79a8a4d9587b3fd62fed2cc67734ec1f519330fefaf
Nov 02 13:00:49.569592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.569554    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl" event={"ID":"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe","Type":"ContainerStarted","Data":"576ed9ff7d101f3e08fed79a8a4d9587b3fd62fed2cc67734ec1f519330fefaf"}
Nov 02 13:00:49.666727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.666692    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6995-76534c"]
Nov 02 13:00:49.672184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.672160    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6995-76534c"
Nov 02 13:00:49.688466 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.688435    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hfcns\""
Nov 02 13:00:49.692446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.692416    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6995-76534c"]
Nov 02 13:00:49.705156 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:49.705108    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f133de5_ca30_497b_a5b3_8eb19ba650e1.slice/crio-a6325a9c77494253e34d84458a61203134cf769c81ba870142426939c67e994c WatchSource:0}: Error finding container a6325a9c77494253e34d84458a61203134cf769c81ba870142426939c67e994c: Status 404 returned error can't find the container with id a6325a9c77494253e34d84458a61203134cf769c81ba870142426939c67e994c
Nov 02 13:00:49.705600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.705581    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"]
Nov 02 13:00:49.774796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.774760    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxv5d\" (UniqueName: \"kubernetes.io/projected/0e72fa3e-a3af-4c81-9bb9-5e59fa96620a-kube-api-access-jxv5d\") pod \"test-curl-6995-76534c\" (UID: \"0e72fa3e-a3af-4c81-9bb9-5e59fa96620a\") " pod="opendatahub/test-curl-6995-76534c"
Nov 02 13:00:49.876430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.876392    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxv5d\" (UniqueName: \"kubernetes.io/projected/0e72fa3e-a3af-4c81-9bb9-5e59fa96620a-kube-api-access-jxv5d\") pod \"test-curl-6995-76534c\" (UID: \"0e72fa3e-a3af-4c81-9bb9-5e59fa96620a\") " pod="opendatahub/test-curl-6995-76534c"
Nov 02 13:00:49.901271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.901241    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxv5d\" (UniqueName: \"kubernetes.io/projected/0e72fa3e-a3af-4c81-9bb9-5e59fa96620a-kube-api-access-jxv5d\") pod \"test-curl-6995-76534c\" (UID: \"0e72fa3e-a3af-4c81-9bb9-5e59fa96620a\") " pod="opendatahub/test-curl-6995-76534c"
Nov 02 13:00:49.987044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:49.987008    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6995-76534c"
Nov 02 13:00:50.153270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:50.153239    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:50.294607 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:00:50.294576    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e72fa3e_a3af_4c81_9bb9_5e59fa96620a.slice/crio-1af684d0275ef19be12932eca50333f2920a7d603c8c6cae5845c721b6c0caed WatchSource:0}: Error finding container 1af684d0275ef19be12932eca50333f2920a7d603c8c6cae5845c721b6c0caed: Status 404 returned error can't find the container with id 1af684d0275ef19be12932eca50333f2920a7d603c8c6cae5845c721b6c0caed
Nov 02 13:00:50.294750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:50.294634    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6995-76534c"]
Nov 02 13:00:50.578960 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:50.578850    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl" event={"ID":"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe","Type":"ContainerStarted","Data":"3b145a61967a40558e5fa6c69acf98e3a008e1d067bbd154f17137d759cfeaee"}
Nov 02 13:00:50.578960 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:50.578921    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"
Nov 02 13:00:50.580870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:50.580840    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6995-76534c" event={"ID":"0e72fa3e-a3af-4c81-9bb9-5e59fa96620a","Type":"ContainerStarted","Data":"3256ae55e813e3e6b1b66f285dc4757fbb15b4fb255203d310403b928ed90a4f"}
Nov 02 13:00:50.581053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:50.580876    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6995-76534c" event={"ID":"0e72fa3e-a3af-4c81-9bb9-5e59fa96620a","Type":"ContainerStarted","Data":"1af684d0275ef19be12932eca50333f2920a7d603c8c6cae5845c721b6c0caed"}
Nov 02 13:00:50.582793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:50.582766    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z" event={"ID":"5f133de5-ca30-497b-a5b3-8eb19ba650e1","Type":"ContainerStarted","Data":"1f1540949921663576520ae0ceb8273c5d93058af93bef21c0fe8b01b5c88fea"}
Nov 02 13:00:50.582938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:50.582822    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"
Nov 02 13:00:50.582938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:50.582845    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z" event={"ID":"5f133de5-ca30-497b-a5b3-8eb19ba650e1","Type":"ContainerStarted","Data":"a6325a9c77494253e34d84458a61203134cf769c81ba870142426939c67e994c"}
Nov 02 13:00:50.623486 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:50.623424    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl" podStartSLOduration=2.623408699 podStartE2EDuration="2.623408699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:50.622669432 +0000 UTC m=+1996.729372265" watchObservedRunningTime="2026-11-02 13:00:50.623408699 +0000 UTC m=+1996.730111549"
Nov 02 13:00:50.666357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:50.666305    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z" podStartSLOduration=2.666291992 podStartE2EDuration="2.666291992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:50.665558726 +0000 UTC m=+1996.772261578" watchObservedRunningTime="2026-11-02 13:00:50.666291992 +0000 UTC m=+1996.772994876"
Nov 02 13:00:50.709223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:50.709158    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6995-76534c" podStartSLOduration=1.709142052 podStartE2EDuration="1.709142052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:00:50.70611407 +0000 UTC m=+1996.812816921" watchObservedRunningTime="2026-11-02 13:00:50.709142052 +0000 UTC m=+1996.815844903"
Nov 02 13:00:51.606684 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:51.606655    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6995-76534c"]
Nov 02 13:00:51.610917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:51.610889    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6995-76534c"]
Nov 02 13:00:51.752080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:51.752045    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:52.596961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:52.596919    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6995-76534c" podUID="0e72fa3e-a3af-4c81-9bb9-5e59fa96620a" containerName="test-curl-6995-76534c" containerID="cri-o://3256ae55e813e3e6b1b66f285dc4757fbb15b4fb255203d310403b928ed90a4f" gracePeriod=2
Nov 02 13:00:53.014635 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:53.014603    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:53.145195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:53.145161    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:53.292423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:53.292305    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:53.708822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:53.708785    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:53.845081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:53.845044    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:54.053066 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:54.038643    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:54.182149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:54.182111    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:54.718934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:54.718894    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:54.835840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:54.835811    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6995-76534c"
Nov 02 13:00:54.957971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:54.957943    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e72fa3e-a3af-4c81-9bb9-5e59fa96620a-kube-api-access-jxv5d\" (UniqueName: \"kubernetes.io/projected/0e72fa3e-a3af-4c81-9bb9-5e59fa96620a-kube-api-access-jxv5d\") pod \"0e72fa3e-a3af-4c81-9bb9-5e59fa96620a\" (UID: \"0e72fa3e-a3af-4c81-9bb9-5e59fa96620a\") "
Nov 02 13:00:54.960823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:54.960785    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e72fa3e-a3af-4c81-9bb9-5e59fa96620a-kube-api-access-jxv5d" pod "0e72fa3e-a3af-4c81-9bb9-5e59fa96620a" (UID: "0e72fa3e-a3af-4c81-9bb9-5e59fa96620a"). InnerVolumeSpecName "kube-api-access-jxv5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:00:55.060392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:55.060357    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxv5d\" (UniqueName: \"kubernetes.io/projected/0e72fa3e-a3af-4c81-9bb9-5e59fa96620a-kube-api-access-jxv5d\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:00:55.249196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:55.249096    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:55.385682 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:55.385649    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:55.524075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:55.523365    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:55.623417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:55.623381    5879 generic.go:356] "Generic (PLEG): container finished" podID="0e72fa3e-a3af-4c81-9bb9-5e59fa96620a" containerID="3256ae55e813e3e6b1b66f285dc4757fbb15b4fb255203d310403b928ed90a4f" exitCode=137
Nov 02 13:00:55.623601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:55.623488    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6995-76534c"
Nov 02 13:00:55.623601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:55.623547    5879 scope.go:122] "RemoveContainer" containerID="3256ae55e813e3e6b1b66f285dc4757fbb15b4fb255203d310403b928ed90a4f"
Nov 02 13:00:55.637392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:55.637368    5879 scope.go:122] "RemoveContainer" containerID="3256ae55e813e3e6b1b66f285dc4757fbb15b4fb255203d310403b928ed90a4f"
Nov 02 13:00:55.637754 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:55.637733    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3256ae55e813e3e6b1b66f285dc4757fbb15b4fb255203d310403b928ed90a4f\": container with ID starting with 3256ae55e813e3e6b1b66f285dc4757fbb15b4fb255203d310403b928ed90a4f not found: ID does not exist" containerID="3256ae55e813e3e6b1b66f285dc4757fbb15b4fb255203d310403b928ed90a4f"
Nov 02 13:00:55.637823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:55.637768    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3256ae55e813e3e6b1b66f285dc4757fbb15b4fb255203d310403b928ed90a4f"} err="failed to get container status \"3256ae55e813e3e6b1b66f285dc4757fbb15b4fb255203d310403b928ed90a4f\": rpc error: code = NotFound desc = could not find container \"3256ae55e813e3e6b1b66f285dc4757fbb15b4fb255203d310403b928ed90a4f\": container with ID starting with 3256ae55e813e3e6b1b66f285dc4757fbb15b4fb255203d310403b928ed90a4f not found: ID does not exist"
Nov 02 13:00:55.647672 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:00:55.647629    5879 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6995-76534c\" is forbidden: User \"system:node:ip-10-0-134-229.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-229.ec2.internal' and this object" podUID="0e72fa3e-a3af-4c81-9bb9-5e59fa96620a" pod="opendatahub/test-curl-6995-76534c"
Nov 02 13:00:56.017510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:56.017478    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e72fa3e-a3af-4c81-9bb9-5e59fa96620a" path="/var/lib/kubelet/pods/0e72fa3e-a3af-4c81-9bb9-5e59fa96620a/volumes"
Nov 02 13:00:56.762967 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:56.762934    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:56.885549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:56.885519    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:57.051032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:57.050923    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:57.175340 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:57.175299    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:57.756703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:57.756664    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:57.948018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:57.942580    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:59.160563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:59.160526    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:00:59.295338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:00:59.295286    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:00.287875 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.287843    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:00.425316 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.425280    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"]
Nov 02 13:01:00.425923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.425910    5879 cpu_manager.go:414] "containerMap: removing container" podUID="0e72fa3e-a3af-4c81-9bb9-5e59fa96620a" containerName="test-curl-6995-76534c"
Nov 02 13:01:00.425969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.425925    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e72fa3e-a3af-4c81-9bb9-5e59fa96620a" containerName="test-curl-6995-76534c"
Nov 02 13:01:00.426065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.426054    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e72fa3e-a3af-4c81-9bb9-5e59fa96620a" containerName="test-curl-6995-76534c"
Nov 02 13:01:00.430943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.430921    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"
Nov 02 13:01:00.433505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.433484    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:00.449952 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.449923    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"]
Nov 02 13:01:00.459578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.459548    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-740f09\""
Nov 02 13:01:00.459578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.459561    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-740f09-dockercfg-zbkcw\""
Nov 02 13:01:00.565108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.543084    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfnft\" (UniqueName: \"kubernetes.io/projected/8bee752b-e2a7-4348-b924-8615b18822d7-kube-api-access-wfnft\") pod \"payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn\" (UID: \"8bee752b-e2a7-4348-b924-8615b18822d7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"
Nov 02 13:01:00.567708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.567648    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bee752b-e2a7-4348-b924-8615b18822d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn\" (UID: \"8bee752b-e2a7-4348-b924-8615b18822d7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"
Nov 02 13:01:00.604959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.604912    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"]
Nov 02 13:01:00.610654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.610624    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"]
Nov 02 13:01:00.611066 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.610942    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:01:00.669495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.669461    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfnft\" (UniqueName: \"kubernetes.io/projected/8bee752b-e2a7-4348-b924-8615b18822d7-kube-api-access-wfnft\") pod \"payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn\" (UID: \"8bee752b-e2a7-4348-b924-8615b18822d7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"
Nov 02 13:01:00.669708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.669645    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bee752b-e2a7-4348-b924-8615b18822d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn\" (UID: \"8bee752b-e2a7-4348-b924-8615b18822d7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"
Nov 02 13:01:00.670342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.670317    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bee752b-e2a7-4348-b924-8615b18822d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn\" (UID: \"8bee752b-e2a7-4348-b924-8615b18822d7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"
Nov 02 13:01:00.694597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.694562    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfnft\" (UniqueName: \"kubernetes.io/projected/8bee752b-e2a7-4348-b924-8615b18822d7-kube-api-access-wfnft\") pod \"payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn\" (UID: \"8bee752b-e2a7-4348-b924-8615b18822d7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"
Nov 02 13:01:00.745468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.745435    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"
Nov 02 13:01:00.771356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.771317    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmnr\" (UniqueName: \"kubernetes.io/projected/f44570e6-9d69-418d-8a2a-8dcbe46857f2-kube-api-access-plmnr\") pod \"payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8\" (UID: \"f44570e6-9d69-418d-8a2a-8dcbe46857f2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:01:00.771504 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.771460    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f44570e6-9d69-418d-8a2a-8dcbe46857f2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8\" (UID: \"f44570e6-9d69-418d-8a2a-8dcbe46857f2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:01:00.789459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.789426    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:00.873322 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.873287    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmnr\" (UniqueName: \"kubernetes.io/projected/f44570e6-9d69-418d-8a2a-8dcbe46857f2-kube-api-access-plmnr\") pod \"payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8\" (UID: \"f44570e6-9d69-418d-8a2a-8dcbe46857f2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:01:00.873535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.873389    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f44570e6-9d69-418d-8a2a-8dcbe46857f2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8\" (UID: \"f44570e6-9d69-418d-8a2a-8dcbe46857f2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:01:00.874249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.874223    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f44570e6-9d69-418d-8a2a-8dcbe46857f2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8\" (UID: \"f44570e6-9d69-418d-8a2a-8dcbe46857f2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:01:00.898235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.898203    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmnr\" (UniqueName: \"kubernetes.io/projected/f44570e6-9d69-418d-8a2a-8dcbe46857f2-kube-api-access-plmnr\") pod \"payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8\" (UID: \"f44570e6-9d69-418d-8a2a-8dcbe46857f2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:01:00.925936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:00.925911    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:01:01.054503 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:01:01.054471    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bee752b_e2a7_4348_b924_8615b18822d7.slice/crio-3073934cad934f1ef63add0f903531958558bc907950d1591f7053a3b03950e6 WatchSource:0}: Error finding container 3073934cad934f1ef63add0f903531958558bc907950d1591f7053a3b03950e6: Status 404 returned error can't find the container with id 3073934cad934f1ef63add0f903531958558bc907950d1591f7053a3b03950e6
Nov 02 13:01:01.055892 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:01.055862    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"]
Nov 02 13:01:01.232966 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:01:01.232935    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44570e6_9d69_418d_8a2a_8dcbe46857f2.slice/crio-9ac0c0735c3d1b44288fa89df98c78e01fbc110d25b92d8f9ab7d7c76e560aaf WatchSource:0}: Error finding container 9ac0c0735c3d1b44288fa89df98c78e01fbc110d25b92d8f9ab7d7c76e560aaf: Status 404 returned error can't find the container with id 9ac0c0735c3d1b44288fa89df98c78e01fbc110d25b92d8f9ab7d7c76e560aaf
Nov 02 13:01:01.232966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:01.232957    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"]
Nov 02 13:01:01.593598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:01.593516    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"
Nov 02 13:01:01.593598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:01.593581    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"
Nov 02 13:01:01.672725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:01.672687    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8" event={"ID":"f44570e6-9d69-418d-8a2a-8dcbe46857f2","Type":"ContainerStarted","Data":"a374ab583818a172e51deab2cc508570f1062a60e0236ec7ef6bf10a5d9a16a5"}
Nov 02 13:01:01.672933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:01.672763    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:01:01.672933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:01.672790    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8" event={"ID":"f44570e6-9d69-418d-8a2a-8dcbe46857f2","Type":"ContainerStarted","Data":"9ac0c0735c3d1b44288fa89df98c78e01fbc110d25b92d8f9ab7d7c76e560aaf"}
Nov 02 13:01:01.674571 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:01.674539    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn" event={"ID":"8bee752b-e2a7-4348-b924-8615b18822d7","Type":"ContainerStarted","Data":"3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796"}
Nov 02 13:01:01.674697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:01.674594    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn" event={"ID":"8bee752b-e2a7-4348-b924-8615b18822d7","Type":"ContainerStarted","Data":"3073934cad934f1ef63add0f903531958558bc907950d1591f7053a3b03950e6"}
Nov 02 13:01:01.674697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:01.674623    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"
Nov 02 13:01:01.733984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:01.733929    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8" podStartSLOduration=1.733915513 podStartE2EDuration="1.733915513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:01.731047165 +0000 UTC m=+2007.837750015" watchObservedRunningTime="2026-11-02 13:01:01.733915513 +0000 UTC m=+2007.840618357"
Nov 02 13:01:01.775656 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:01.775606    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn" podStartSLOduration=1.775587451 podStartE2EDuration="1.775587451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:01.773140833 +0000 UTC m=+2007.879843690" watchObservedRunningTime="2026-11-02 13:01:01.775587451 +0000 UTC m=+2007.882290304"
Nov 02 13:01:02.477762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:02.477721    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:04.159277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.159242    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:04.455172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.455079    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99" podUID="70bda333-74a7-4dad-8e23-7c98eb74e540" containerName="main" containerID="cri-o://79c52a6f15e05346b9f6bb3d9e051b38b586b7e2a2f2d2dec0fbc9a9ee26e059" gracePeriod=45
Nov 02 13:01:04.700559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.700525    5879 generic.go:356] "Generic (PLEG): container finished" podID="70bda333-74a7-4dad-8e23-7c98eb74e540" containerID="79c52a6f15e05346b9f6bb3d9e051b38b586b7e2a2f2d2dec0fbc9a9ee26e059" exitCode=0
Nov 02 13:01:04.700735 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.700592    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99" event={"ID":"70bda333-74a7-4dad-8e23-7c98eb74e540","Type":"ContainerDied","Data":"79c52a6f15e05346b9f6bb3d9e051b38b586b7e2a2f2d2dec0fbc9a9ee26e059"}
Nov 02 13:01:04.743789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.743757    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 13:01:04.825185 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.825155    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-model-cache\") pod \"70bda333-74a7-4dad-8e23-7c98eb74e540\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") "
Nov 02 13:01:04.825400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.825268    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bda333-74a7-4dad-8e23-7c98eb74e540-kube-api-access-mvq5j\" (UniqueName: \"kubernetes.io/projected/70bda333-74a7-4dad-8e23-7c98eb74e540-kube-api-access-mvq5j\") pod \"70bda333-74a7-4dad-8e23-7c98eb74e540\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") "
Nov 02 13:01:04.825400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.825294    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-dshm\") pod \"70bda333-74a7-4dad-8e23-7c98eb74e540\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") "
Nov 02 13:01:04.825400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.825319    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70bda333-74a7-4dad-8e23-7c98eb74e540-tls-certs\" (UniqueName: \"kubernetes.io/secret/70bda333-74a7-4dad-8e23-7c98eb74e540-tls-certs\") pod \"70bda333-74a7-4dad-8e23-7c98eb74e540\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") "
Nov 02 13:01:04.825400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.825371    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-home\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-home\") pod \"70bda333-74a7-4dad-8e23-7c98eb74e540\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") "
Nov 02 13:01:04.825644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.825436    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-tmp-dir\") pod \"70bda333-74a7-4dad-8e23-7c98eb74e540\" (UID: \"70bda333-74a7-4dad-8e23-7c98eb74e540\") "
Nov 02 13:01:04.825644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.825458    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-model-cache" pod "70bda333-74a7-4dad-8e23-7c98eb74e540" (UID: "70bda333-74a7-4dad-8e23-7c98eb74e540"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:04.826042 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.825971    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-tmp-dir" pod "70bda333-74a7-4dad-8e23-7c98eb74e540" (UID: "70bda333-74a7-4dad-8e23-7c98eb74e540"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:04.826195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.826048    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:04.826195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.826158    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-home" pod "70bda333-74a7-4dad-8e23-7c98eb74e540" (UID: "70bda333-74a7-4dad-8e23-7c98eb74e540"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:04.827745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.827723    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bda333-74a7-4dad-8e23-7c98eb74e540-kube-api-access-mvq5j" pod "70bda333-74a7-4dad-8e23-7c98eb74e540" (UID: "70bda333-74a7-4dad-8e23-7c98eb74e540"). InnerVolumeSpecName "kube-api-access-mvq5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:01:04.827978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.827961    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-dshm" pod "70bda333-74a7-4dad-8e23-7c98eb74e540" (UID: "70bda333-74a7-4dad-8e23-7c98eb74e540"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:04.827978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.827967    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70bda333-74a7-4dad-8e23-7c98eb74e540-tls-certs" pod "70bda333-74a7-4dad-8e23-7c98eb74e540" (UID: "70bda333-74a7-4dad-8e23-7c98eb74e540"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:01:04.927753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.927722    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:04.927753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.927754    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvq5j\" (UniqueName: \"kubernetes.io/projected/70bda333-74a7-4dad-8e23-7c98eb74e540-kube-api-access-mvq5j\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:04.927986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.927766    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:04.927986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.927778    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bda333-74a7-4dad-8e23-7c98eb74e540-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:04.927986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:04.927793    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bda333-74a7-4dad-8e23-7c98eb74e540-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:05.708887 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:05.708853    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"
Nov 02 13:01:05.708887 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:05.708872    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99" event={"ID":"70bda333-74a7-4dad-8e23-7c98eb74e540","Type":"ContainerDied","Data":"b81fcdb532542c1c207f63f9cefba29c986ff7237a452879324e1c9a691651a6"}
Nov 02 13:01:05.709502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:05.708941    5879 scope.go:122] "RemoveContainer" containerID="79c52a6f15e05346b9f6bb3d9e051b38b586b7e2a2f2d2dec0fbc9a9ee26e059"
Nov 02 13:01:05.766056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:05.766021    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"]
Nov 02 13:01:05.767962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:05.767932    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:05.779941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:05.779909    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1aa18c/auth-test-model-1aa18c-kserve-68b4b87d4f-qxt99"]
Nov 02 13:01:06.018716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:06.018636    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bda333-74a7-4dad-8e23-7c98eb74e540" path="/var/lib/kubelet/pods/70bda333-74a7-4dad-8e23-7c98eb74e540/volumes"
Nov 02 13:01:07.312566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:07.312530    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:08.897107 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:08.895276    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:10.001445 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.001413    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"]
Nov 02 13:01:10.002370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.002352    5879 cpu_manager.go:414] "containerMap: removing container" podUID="70bda333-74a7-4dad-8e23-7c98eb74e540" containerName="main"
Nov 02 13:01:10.002433 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.002373    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bda333-74a7-4dad-8e23-7c98eb74e540" containerName="main"
Nov 02 13:01:10.002548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.002534    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bda333-74a7-4dad-8e23-7c98eb74e540" containerName="main"
Nov 02 13:01:10.007550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.007530    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.034143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.034113    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-7aca74\"/\"kube-root-ca.crt\""
Nov 02 13:01:10.034143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.034130    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-7aca74\"/\"openshift-service-ca.crt\""
Nov 02 13:01:10.037128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.037096    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"]
Nov 02 13:01:10.044074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.044047    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-7aca74\"/\"default-dockercfg-mgx4s\""
Nov 02 13:01:10.044240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.044047    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-7aca74\"/\"noten-model-7aca74-kserve-self-signed-certs\""
Nov 02 13:01:10.074950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.074910    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:10.097647 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.097601    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-tmp-dir\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.097805 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.097703    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-home\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.097884 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.097866    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-dshm\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.097942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.097925    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88w8f\" (UniqueName: \"kubernetes.io/projected/172470fc-484a-449c-b67a-5ada842b4310-kube-api-access-88w8f\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.098033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.097985    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-model-cache\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.098262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.098237    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172470fc-484a-449c-b67a-5ada842b4310-tls-certs\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.199160 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.199124    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-dshm\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.199377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.199180    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88w8f\" (UniqueName: \"kubernetes.io/projected/172470fc-484a-449c-b67a-5ada842b4310-kube-api-access-88w8f\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.199377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.199223    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-model-cache\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.199377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.199271    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172470fc-484a-449c-b67a-5ada842b4310-tls-certs\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.199377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.199341    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-tmp-dir\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.199617 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.199394    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-home\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.200131 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.199789    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-home\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.200131 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.199850    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-model-cache\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.200131 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.200087    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-tmp-dir\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.202256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.202223    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-dshm\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.203166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.203137    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172470fc-484a-449c-b67a-5ada842b4310-tls-certs\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.228214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.228177    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88w8f\" (UniqueName: \"kubernetes.io/projected/172470fc-484a-449c-b67a-5ada842b4310-kube-api-access-88w8f\") pod \"noten-model-7aca74-kserve-589f69d869-8mjlj\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") " pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.334449 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.334364    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:10.746965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.746930    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:10.760987 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.760964    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"]
Nov 02 13:01:10.763657 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:01:10.763625    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod172470fc_484a_449c_b67a_5ada842b4310.slice/crio-704ab79c37a2021443d544627797dcaeca7c5a25fb3d182ab26bb32dc6e02f65 WatchSource:0}: Error finding container 704ab79c37a2021443d544627797dcaeca7c5a25fb3d182ab26bb32dc6e02f65: Status 404 returned error can't find the container with id 704ab79c37a2021443d544627797dcaeca7c5a25fb3d182ab26bb32dc6e02f65
Nov 02 13:01:10.780899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:10.780794    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj" event={"ID":"172470fc-484a-449c-b67a-5ada842b4310","Type":"ContainerStarted","Data":"704ab79c37a2021443d544627797dcaeca7c5a25fb3d182ab26bb32dc6e02f65"}
Nov 02 13:01:11.790122 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:11.790080    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj" event={"ID":"172470fc-484a-449c-b67a-5ada842b4310","Type":"ContainerStarted","Data":"4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d"}
Nov 02 13:01:11.839777 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:11.839704    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj" podStartSLOduration=2.83968378 podStartE2EDuration="2.83968378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:11.837896528 +0000 UTC m=+2017.944599380" watchObservedRunningTime="2026-11-02 13:01:11.83968378 +0000 UTC m=+2017.946386637"
Nov 02 13:01:12.334501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:12.334456    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:12.570008 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:12.569956    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:12.686242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:12.686213    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:01:12.686423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:12.686283    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"
Nov 02 13:01:14.110325 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:14.110288    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:14.256495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:14.256462    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:14.490377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:14.490341    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:14.826212 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:01:14.826129    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:01:15.736140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:15.736104    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:17.383505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:17.383471    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:18.977668 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:18.977632    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"]
Nov 02 13:01:19.088691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:19.088655    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:19.245405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:19.245304    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:19.408895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:19.408843    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:19.580266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:19.580182    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:19.700971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:19.700945    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:20.746451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:20.746414    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:21.148020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.147957    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"]
Nov 02 13:01:21.152910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.152884    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.171301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.171276    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ecec44\"/\"kube-root-ca.crt\""
Nov 02 13:01:21.178686 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.175784    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"]
Nov 02 13:01:21.185244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.185213    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-ecec44\"/\"openshift-service-ca.crt\""
Nov 02 13:01:21.185244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.185220    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ecec44\"/\"autores-model-ecec44-kserve-self-signed-certs\""
Nov 02 13:01:21.185525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.185274    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-ecec44\"/\"default-dockercfg-7529k\""
Nov 02 13:01:21.225716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.225679    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:21.278928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.278884    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-model-cache\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.279146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.278934    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps9s8\" (UniqueName: \"kubernetes.io/projected/cce2991e-3460-4dbf-874a-fcbd85670d2e-kube-api-access-ps9s8\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.279215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.279195    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-dshm\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.279271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.279262    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cce2991e-3460-4dbf-874a-fcbd85670d2e-tls-certs\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.279370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.279352    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-home\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.279479 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.279458    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-tmp-dir\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.383014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.380735    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-dshm\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.383014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.380838    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cce2991e-3460-4dbf-874a-fcbd85670d2e-tls-certs\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.383014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.380901    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-home\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.383014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.381029    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-tmp-dir\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.383014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.381081    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-model-cache\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.383014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.381107    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps9s8\" (UniqueName: \"kubernetes.io/projected/cce2991e-3460-4dbf-874a-fcbd85670d2e-kube-api-access-ps9s8\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.383493 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.383408    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-home\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.383880 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.383857    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-tmp-dir\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.384085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.383895    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-model-cache\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.386035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.385286    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-dshm\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.388789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.388739    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cce2991e-3460-4dbf-874a-fcbd85670d2e-tls-certs\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.423094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.416066    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps9s8\" (UniqueName: \"kubernetes.io/projected/cce2991e-3460-4dbf-874a-fcbd85670d2e-kube-api-access-ps9s8\") pod \"autores-model-ecec44-kserve-8645bc99d9-czhjd\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") " pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.471628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.471587    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:21.791080 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:01:21.791045    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcce2991e_3460_4dbf_874a_fcbd85670d2e.slice/crio-c56c93bf06d8401a57f26577a20c582f266e0335c53ec931a23f8a962d837b7c WatchSource:0}: Error finding container c56c93bf06d8401a57f26577a20c582f266e0335c53ec931a23f8a962d837b7c: Status 404 returned error can't find the container with id c56c93bf06d8401a57f26577a20c582f266e0335c53ec931a23f8a962d837b7c
Nov 02 13:01:21.792847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.792820    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"]
Nov 02 13:01:21.862199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.862165    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd" event={"ID":"cce2991e-3460-4dbf-874a-fcbd85670d2e","Type":"ContainerStarted","Data":"be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125"}
Nov 02 13:01:21.862377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.862216    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd" event={"ID":"cce2991e-3460-4dbf-874a-fcbd85670d2e","Type":"ContainerStarted","Data":"c56c93bf06d8401a57f26577a20c582f266e0335c53ec931a23f8a962d837b7c"}
Nov 02 13:01:21.914624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:21.914572    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd" podStartSLOduration=0.914556237 podStartE2EDuration="914.556237ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:21.911021936 +0000 UTC m=+2028.017724787" watchObservedRunningTime="2026-11-02 13:01:21.914556237 +0000 UTC m=+2028.021259088"
Nov 02 13:01:22.412890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:22.412854    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:22.472266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:22.472235    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:23.974629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:23.974580    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:24.693179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:24.693127    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:25.513409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:25.513342    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:25.650465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:25.650432    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:25.960882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:25.960841    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:27.239813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:27.239778    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:28.947408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:28.947376    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:30.643240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:30.643205    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:31.471958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:31.471918    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:31.486702 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:31.486675    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:31.946130 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:31.946104    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:32.099192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:32.099150    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:32.372326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:32.372285    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:33.354104 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:33.354064    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:33.893296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:33.893255    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:34.000693 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.000628    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj" podUID="172470fc-484a-449c-b67a-5ada842b4310" containerName="main" containerID="cri-o://4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d" gracePeriod=45
Nov 02 13:01:34.117394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.117293    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:34.340645 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.340617    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:34.479210 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.479177    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/172470fc-484a-449c-b67a-5ada842b4310-tls-certs\" (UniqueName: \"kubernetes.io/secret/172470fc-484a-449c-b67a-5ada842b4310-tls-certs\") pod \"172470fc-484a-449c-b67a-5ada842b4310\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") "
Nov 02 13:01:34.479210 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.479219    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172470fc-484a-449c-b67a-5ada842b4310-kube-api-access-88w8f\" (UniqueName: \"kubernetes.io/projected/172470fc-484a-449c-b67a-5ada842b4310-kube-api-access-88w8f\") pod \"172470fc-484a-449c-b67a-5ada842b4310\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") "
Nov 02 13:01:34.479827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.479357    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-tmp-dir\") pod \"172470fc-484a-449c-b67a-5ada842b4310\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") "
Nov 02 13:01:34.479827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.479458    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-home\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-home\") pod \"172470fc-484a-449c-b67a-5ada842b4310\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") "
Nov 02 13:01:34.479827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.479546    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-dshm\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-dshm\") pod \"172470fc-484a-449c-b67a-5ada842b4310\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") "
Nov 02 13:01:34.479827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.479624    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-model-cache\") pod \"172470fc-484a-449c-b67a-5ada842b4310\" (UID: \"172470fc-484a-449c-b67a-5ada842b4310\") "
Nov 02 13:01:34.479827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.479645    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-tmp-dir" pod "172470fc-484a-449c-b67a-5ada842b4310" (UID: "172470fc-484a-449c-b67a-5ada842b4310"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:34.479827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.479754    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-home" pod "172470fc-484a-449c-b67a-5ada842b4310" (UID: "172470fc-484a-449c-b67a-5ada842b4310"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:34.480147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.479935    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-model-cache" pod "172470fc-484a-449c-b67a-5ada842b4310" (UID: "172470fc-484a-449c-b67a-5ada842b4310"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:34.480346 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.480324    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:34.480399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.480346    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:34.480399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.480359    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:34.482241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.482218    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-dshm" pod "172470fc-484a-449c-b67a-5ada842b4310" (UID: "172470fc-484a-449c-b67a-5ada842b4310"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:34.482383 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.482360    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/172470fc-484a-449c-b67a-5ada842b4310-tls-certs" pod "172470fc-484a-449c-b67a-5ada842b4310" (UID: "172470fc-484a-449c-b67a-5ada842b4310"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:01:34.482444 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.482364    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172470fc-484a-449c-b67a-5ada842b4310-kube-api-access-88w8f" pod "172470fc-484a-449c-b67a-5ada842b4310" (UID: "172470fc-484a-449c-b67a-5ada842b4310"). InnerVolumeSpecName "kube-api-access-88w8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:01:34.582427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.582385    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172470fc-484a-449c-b67a-5ada842b4310-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:34.582427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.582423    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172470fc-484a-449c-b67a-5ada842b4310-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:34.582646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.582439    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88w8f\" (UniqueName: \"kubernetes.io/projected/172470fc-484a-449c-b67a-5ada842b4310-kube-api-access-88w8f\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:34.960547 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.960508    5879 generic.go:356] "Generic (PLEG): container finished" podID="172470fc-484a-449c-b67a-5ada842b4310" containerID="4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d" exitCode=0
Nov 02 13:01:34.960762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.960572    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"
Nov 02 13:01:34.960762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.960592    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj" event={"ID":"172470fc-484a-449c-b67a-5ada842b4310","Type":"ContainerDied","Data":"4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d"}
Nov 02 13:01:34.960762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.960678    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj" event={"ID":"172470fc-484a-449c-b67a-5ada842b4310","Type":"ContainerDied","Data":"704ab79c37a2021443d544627797dcaeca7c5a25fb3d182ab26bb32dc6e02f65"}
Nov 02 13:01:34.960762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.960701    5879 scope.go:122] "RemoveContainer" containerID="4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d"
Nov 02 13:01:34.973361 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.973335    5879 scope.go:122] "RemoveContainer" containerID="4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d"
Nov 02 13:01:34.973657 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:01:34.973640    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d\": container with ID starting with 4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d not found: ID does not exist" containerID="4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d"
Nov 02 13:01:34.973714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:34.973667    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d"} err="failed to get container status \"4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d\": rpc error: code = NotFound desc = could not find container \"4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d\": container with ID starting with 4e4d418264f77b6e39824ce9900948c6f16c6c25a287d287b6795bd78592a47d not found: ID does not exist"
Nov 02 13:01:35.025366 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:35.025335    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"]
Nov 02 13:01:35.038777 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:35.038749    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-7aca74/noten-model-7aca74-kserve-589f69d869-8mjlj"]
Nov 02 13:01:35.681747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:35.681714    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:36.023174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:36.023139    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="172470fc-484a-449c-b67a-5ada842b4310" path="/var/lib/kubelet/pods/172470fc-484a-449c-b67a-5ada842b4310/volumes"
Nov 02 13:01:37.375026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:37.374969    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:37.584047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:37.583066    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:39.364121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.364066    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:39.553117 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.553073    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"]
Nov 02 13:01:39.553775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.553755    5879 cpu_manager.go:414] "containerMap: removing container" podUID="172470fc-484a-449c-b67a-5ada842b4310" containerName="main"
Nov 02 13:01:39.553892 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.553775    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="172470fc-484a-449c-b67a-5ada842b4310" containerName="main"
Nov 02 13:01:39.554030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.554013    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="172470fc-484a-449c-b67a-5ada842b4310" containerName="main"
Nov 02 13:01:39.557741 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.557720    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"
Nov 02 13:01:39.585094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.585060    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-00d004-dockercfg-4z2mc\""
Nov 02 13:01:39.596348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.596319    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:39.599086 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.599063    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-00d004\""
Nov 02 13:01:39.604603 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.604547    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"]
Nov 02 13:01:39.609410 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.609385    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"]
Nov 02 13:01:39.615026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.614960    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"
Nov 02 13:01:39.640232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.640199    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"]
Nov 02 13:01:39.659526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.659208    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwrsd\" (UniqueName: \"kubernetes.io/projected/bfb8420b-2f5d-4d05-81c5-850259014a61-kube-api-access-vwrsd\") pod \"payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj\" (UID: \"bfb8420b-2f5d-4d05-81c5-850259014a61\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"
Nov 02 13:01:39.659526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.659392    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb8420b-2f5d-4d05-81c5-850259014a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj\" (UID: \"bfb8420b-2f5d-4d05-81c5-850259014a61\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"
Nov 02 13:01:39.764017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.761675    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwrsd\" (UniqueName: \"kubernetes.io/projected/bfb8420b-2f5d-4d05-81c5-850259014a61-kube-api-access-vwrsd\") pod \"payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj\" (UID: \"bfb8420b-2f5d-4d05-81c5-850259014a61\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"
Nov 02 13:01:39.764017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.761789    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fbdj\" (UniqueName: \"kubernetes.io/projected/aa872acf-f31b-45f6-bd50-629153ec892b-kube-api-access-8fbdj\") pod \"payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7\" (UID: \"aa872acf-f31b-45f6-bd50-629153ec892b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"
Nov 02 13:01:39.764017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.761835    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa872acf-f31b-45f6-bd50-629153ec892b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7\" (UID: \"aa872acf-f31b-45f6-bd50-629153ec892b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"
Nov 02 13:01:39.764017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.761879    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb8420b-2f5d-4d05-81c5-850259014a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj\" (UID: \"bfb8420b-2f5d-4d05-81c5-850259014a61\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"
Nov 02 13:01:39.764017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.762767    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb8420b-2f5d-4d05-81c5-850259014a61-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj\" (UID: \"bfb8420b-2f5d-4d05-81c5-850259014a61\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"
Nov 02 13:01:39.792230 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.792203    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwrsd\" (UniqueName: \"kubernetes.io/projected/bfb8420b-2f5d-4d05-81c5-850259014a61-kube-api-access-vwrsd\") pod \"payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj\" (UID: \"bfb8420b-2f5d-4d05-81c5-850259014a61\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"
Nov 02 13:01:39.863092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.863050    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fbdj\" (UniqueName: \"kubernetes.io/projected/aa872acf-f31b-45f6-bd50-629153ec892b-kube-api-access-8fbdj\") pod \"payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7\" (UID: \"aa872acf-f31b-45f6-bd50-629153ec892b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"
Nov 02 13:01:39.863304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.863113    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa872acf-f31b-45f6-bd50-629153ec892b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7\" (UID: \"aa872acf-f31b-45f6-bd50-629153ec892b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"
Nov 02 13:01:39.863841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.863821    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa872acf-f31b-45f6-bd50-629153ec892b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7\" (UID: \"aa872acf-f31b-45f6-bd50-629153ec892b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"
Nov 02 13:01:39.871734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.871671    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"
Nov 02 13:01:39.888813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.888786    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fbdj\" (UniqueName: \"kubernetes.io/projected/aa872acf-f31b-45f6-bd50-629153ec892b-kube-api-access-8fbdj\") pod \"payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7\" (UID: \"aa872acf-f31b-45f6-bd50-629153ec892b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"
Nov 02 13:01:39.940183 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:39.939345    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"
Nov 02 13:01:40.181349 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:01:40.181308    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb8420b_2f5d_4d05_81c5_850259014a61.slice/crio-b5efafea981afb96086c4adb9c8a676fd5e7fc0188cec2f44edc8464710b1ef5 WatchSource:0}: Error finding container b5efafea981afb96086c4adb9c8a676fd5e7fc0188cec2f44edc8464710b1ef5: Status 404 returned error can't find the container with id b5efafea981afb96086c4adb9c8a676fd5e7fc0188cec2f44edc8464710b1ef5
Nov 02 13:01:40.183302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:40.183278    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"]
Nov 02 13:01:40.270481 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:01:40.270451    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa872acf_f31b_45f6_bd50_629153ec892b.slice/crio-04b22aec6a0aace7307bb24e372aad5ccfc7ca0b6c2963d8d1cf87561ea4cfdf WatchSource:0}: Error finding container 04b22aec6a0aace7307bb24e372aad5ccfc7ca0b6c2963d8d1cf87561ea4cfdf: Status 404 returned error can't find the container with id 04b22aec6a0aace7307bb24e372aad5ccfc7ca0b6c2963d8d1cf87561ea4cfdf
Nov 02 13:01:40.271920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:40.271898    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"]
Nov 02 13:01:41.016367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:41.016327    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7" event={"ID":"aa872acf-f31b-45f6-bd50-629153ec892b","Type":"ContainerStarted","Data":"5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8"}
Nov 02 13:01:41.016946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:41.016395    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7" event={"ID":"aa872acf-f31b-45f6-bd50-629153ec892b","Type":"ContainerStarted","Data":"04b22aec6a0aace7307bb24e372aad5ccfc7ca0b6c2963d8d1cf87561ea4cfdf"}
Nov 02 13:01:41.016946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:41.016423    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"
Nov 02 13:01:41.019045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:41.018983    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj" event={"ID":"bfb8420b-2f5d-4d05-81c5-850259014a61","Type":"ContainerStarted","Data":"2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b"}
Nov 02 13:01:41.019182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:41.019067    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj" event={"ID":"bfb8420b-2f5d-4d05-81c5-850259014a61","Type":"ContainerStarted","Data":"b5efafea981afb96086c4adb9c8a676fd5e7fc0188cec2f44edc8464710b1ef5"}
Nov 02 13:01:41.019182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:41.019087    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"
Nov 02 13:01:41.061732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:41.061678    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7" podStartSLOduration=2.061662552 podStartE2EDuration="2.061662552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:41.058753757 +0000 UTC m=+2047.165456607" watchObservedRunningTime="2026-11-02 13:01:41.061662552 +0000 UTC m=+2047.168365434"
Nov 02 13:01:41.109549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:41.109486    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj" podStartSLOduration=2.109470635 podStartE2EDuration="2.109470635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:41.107179603 +0000 UTC m=+2047.213882455" watchObservedRunningTime="2026-11-02 13:01:41.109470635 +0000 UTC m=+2047.216173485"
Nov 02 13:01:41.196286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:41.196249    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:42.768188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:42.768152    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:44.060669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:44.060632    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"]
Nov 02 13:01:44.125734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:44.125701    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:44.582249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:44.582217    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:44.701440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:44.701408    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:44.822012 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:01:44.821964    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:01:46.156317 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:46.156278    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:48.104094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:48.104057    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:48.229213 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:48.229182    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:49.509212 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:49.509179    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:49.641197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:49.641159    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:49.770027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:49.769901    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:50.065506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:50.065412    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:50.395048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:50.394959    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:51.344644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:51.344595    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:52.029681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:52.029657    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"
Nov 02 13:01:52.029845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:52.029714    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"
Nov 02 13:01:53.138457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:53.138421    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:53.766312 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:53.766253    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:53.905167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:53.905129    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:54.996513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:54.996477    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:55.557780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:55.557749    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:55.702226 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:55.702187    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:55.820752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:55.820653    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"]
Nov 02 13:01:55.825545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:55.825516    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"
Nov 02 13:01:55.837399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:55.837365    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"]
Nov 02 13:01:55.855638 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:55.855586    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0e422b\""
Nov 02 13:01:55.856713 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:55.856652    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0e422b-dockercfg-84gl4\""
Nov 02 13:01:55.878264 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:55.878224    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"]
Nov 02 13:01:55.878602 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:55.878585    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"
Nov 02 13:01:55.883743 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:55.883716    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"]
Nov 02 13:01:55.904536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:55.904427    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k\" (UID: \"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"
Nov 02 13:01:55.906139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:55.906112    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbhf\" (UniqueName: \"kubernetes.io/projected/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-kube-api-access-4mbhf\") pod \"payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k\" (UID: \"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"
Nov 02 13:01:56.007878 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.007841    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbhf\" (UniqueName: \"kubernetes.io/projected/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-kube-api-access-4mbhf\") pod \"payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k\" (UID: \"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"
Nov 02 13:01:56.008419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.008055    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a117fc4-daac-490d-b03a-4268ee8aaa5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg\" (UID: \"9a117fc4-daac-490d-b03a-4268ee8aaa5c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"
Nov 02 13:01:56.008419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.008171    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k\" (UID: \"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"
Nov 02 13:01:56.008419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.008220    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd685\" (UniqueName: \"kubernetes.io/projected/9a117fc4-daac-490d-b03a-4268ee8aaa5c-kube-api-access-hd685\") pod \"payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg\" (UID: \"9a117fc4-daac-490d-b03a-4268ee8aaa5c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"
Nov 02 13:01:56.008925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.008906    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k\" (UID: \"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"
Nov 02 13:01:56.034280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.034253    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbhf\" (UniqueName: \"kubernetes.io/projected/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-kube-api-access-4mbhf\") pod \"payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k\" (UID: \"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"
Nov 02 13:01:56.109173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.109093    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a117fc4-daac-490d-b03a-4268ee8aaa5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg\" (UID: \"9a117fc4-daac-490d-b03a-4268ee8aaa5c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"
Nov 02 13:01:56.109347 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.109177    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd685\" (UniqueName: \"kubernetes.io/projected/9a117fc4-daac-490d-b03a-4268ee8aaa5c-kube-api-access-hd685\") pod \"payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg\" (UID: \"9a117fc4-daac-490d-b03a-4268ee8aaa5c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"
Nov 02 13:01:56.109869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.109845    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a117fc4-daac-490d-b03a-4268ee8aaa5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg\" (UID: \"9a117fc4-daac-490d-b03a-4268ee8aaa5c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"
Nov 02 13:01:56.135028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.134974    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd685\" (UniqueName: \"kubernetes.io/projected/9a117fc4-daac-490d-b03a-4268ee8aaa5c-kube-api-access-hd685\") pod \"payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg\" (UID: \"9a117fc4-daac-490d-b03a-4268ee8aaa5c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"
Nov 02 13:01:56.185706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.185674    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"
Nov 02 13:01:56.254973 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.254941    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"
Nov 02 13:01:56.499570 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:01:56.499537    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f7ac6a_9176_4d3e_9dee_2ec9abb2b68d.slice/crio-726ecee8f91cdc5b03401c84b513b9c5aec60adfc5e4102e48fafd4949af099e WatchSource:0}: Error finding container 726ecee8f91cdc5b03401c84b513b9c5aec60adfc5e4102e48fafd4949af099e: Status 404 returned error can't find the container with id 726ecee8f91cdc5b03401c84b513b9c5aec60adfc5e4102e48fafd4949af099e
Nov 02 13:01:56.501679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.501657    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"]
Nov 02 13:01:56.578668 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:01:56.578633    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a117fc4_daac_490d_b03a_4268ee8aaa5c.slice/crio-1aa7f269dbec979ecc9f0df6ce39604da22e63baaec9aecb5c3441cce5184e17 WatchSource:0}: Error finding container 1aa7f269dbec979ecc9f0df6ce39604da22e63baaec9aecb5c3441cce5184e17: Status 404 returned error can't find the container with id 1aa7f269dbec979ecc9f0df6ce39604da22e63baaec9aecb5c3441cce5184e17
Nov 02 13:01:56.579659 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.579633    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"]
Nov 02 13:01:56.752245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:56.752136    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:57.136260 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:57.136218    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg" event={"ID":"9a117fc4-daac-490d-b03a-4268ee8aaa5c","Type":"ContainerStarted","Data":"6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b"}
Nov 02 13:01:57.136734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:57.136317    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg" event={"ID":"9a117fc4-daac-490d-b03a-4268ee8aaa5c","Type":"ContainerStarted","Data":"1aa7f269dbec979ecc9f0df6ce39604da22e63baaec9aecb5c3441cce5184e17"}
Nov 02 13:01:57.136734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:57.136471    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"
Nov 02 13:01:57.138086 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:57.138057    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k" event={"ID":"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d","Type":"ContainerStarted","Data":"f19a5855a6635380af824e664f39ab635f77191a56c77254aae93f4cba4ac52f"}
Nov 02 13:01:57.138241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:57.138110    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k" event={"ID":"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d","Type":"ContainerStarted","Data":"726ecee8f91cdc5b03401c84b513b9c5aec60adfc5e4102e48fafd4949af099e"}
Nov 02 13:01:57.138241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:57.138133    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"
Nov 02 13:01:57.183196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:57.183147    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg" podStartSLOduration=2.18313261 podStartE2EDuration="2.18313261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:57.179878262 +0000 UTC m=+2063.286581119" watchObservedRunningTime="2026-11-02 13:01:57.18313261 +0000 UTC m=+2063.289835486"
Nov 02 13:01:57.227594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:57.227535    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k" podStartSLOduration=2.227515962 podStartE2EDuration="2.227515962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:01:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:01:57.2244136 +0000 UTC m=+2063.331116442" watchObservedRunningTime="2026-11-02 13:01:57.227515962 +0000 UTC m=+2063.334218813"
Nov 02 13:01:58.372342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:58.372303    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:58.911437 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:58.911403    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:01:59.088877 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.088843    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd" podUID="cce2991e-3460-4dbf-874a-fcbd85670d2e" containerName="main" containerID="cri-o://be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125" gracePeriod=45
Nov 02 13:01:59.363415 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.363385    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:01:59.455146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.455053    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce2991e-3460-4dbf-874a-fcbd85670d2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/cce2991e-3460-4dbf-874a-fcbd85670d2e-tls-certs\") pod \"cce2991e-3460-4dbf-874a-fcbd85670d2e\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") "
Nov 02 13:01:59.455146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.455096    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-tmp-dir\") pod \"cce2991e-3460-4dbf-874a-fcbd85670d2e\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") "
Nov 02 13:01:59.455679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.455148    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-model-cache\") pod \"cce2991e-3460-4dbf-874a-fcbd85670d2e\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") "
Nov 02 13:01:59.455679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.455252    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-dshm\") pod \"cce2991e-3460-4dbf-874a-fcbd85670d2e\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") "
Nov 02 13:01:59.455679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.455321    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce2991e-3460-4dbf-874a-fcbd85670d2e-kube-api-access-ps9s8\" (UniqueName: \"kubernetes.io/projected/cce2991e-3460-4dbf-874a-fcbd85670d2e-kube-api-access-ps9s8\") pod \"cce2991e-3460-4dbf-874a-fcbd85670d2e\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") "
Nov 02 13:01:59.455679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.455456    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-tmp-dir" pod "cce2991e-3460-4dbf-874a-fcbd85670d2e" (UID: "cce2991e-3460-4dbf-874a-fcbd85670d2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:59.455679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.455475    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-model-cache" pod "cce2991e-3460-4dbf-874a-fcbd85670d2e" (UID: "cce2991e-3460-4dbf-874a-fcbd85670d2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:59.455679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.455524    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-home\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-home\") pod \"cce2991e-3460-4dbf-874a-fcbd85670d2e\" (UID: \"cce2991e-3460-4dbf-874a-fcbd85670d2e\") "
Nov 02 13:01:59.456077 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.455840    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-home" pod "cce2991e-3460-4dbf-874a-fcbd85670d2e" (UID: "cce2991e-3460-4dbf-874a-fcbd85670d2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:59.456383 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.456364    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:59.456418 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.456391    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:59.456418 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.456409    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:59.457893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.457812    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce2991e-3460-4dbf-874a-fcbd85670d2e-kube-api-access-ps9s8" pod "cce2991e-3460-4dbf-874a-fcbd85670d2e" (UID: "cce2991e-3460-4dbf-874a-fcbd85670d2e"). InnerVolumeSpecName "kube-api-access-ps9s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:01:59.458056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.457918    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce2991e-3460-4dbf-874a-fcbd85670d2e-tls-certs" pod "cce2991e-3460-4dbf-874a-fcbd85670d2e" (UID: "cce2991e-3460-4dbf-874a-fcbd85670d2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:01:59.458056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.457955    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-dshm" pod "cce2991e-3460-4dbf-874a-fcbd85670d2e" (UID: "cce2991e-3460-4dbf-874a-fcbd85670d2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:01:59.558173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.558127    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cce2991e-3460-4dbf-874a-fcbd85670d2e-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:59.558173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.558162    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cce2991e-3460-4dbf-874a-fcbd85670d2e-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:01:59.558173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:01:59.558177    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps9s8\" (UniqueName: \"kubernetes.io/projected/cce2991e-3460-4dbf-874a-fcbd85670d2e-kube-api-access-ps9s8\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:02:00.171929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:00.171893    5879 generic.go:356] "Generic (PLEG): container finished" podID="cce2991e-3460-4dbf-874a-fcbd85670d2e" containerID="be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125" exitCode=0
Nov 02 13:02:00.172143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:00.171937    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd" event={"ID":"cce2991e-3460-4dbf-874a-fcbd85670d2e","Type":"ContainerDied","Data":"be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125"}
Nov 02 13:02:00.172143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:00.171987    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd" event={"ID":"cce2991e-3460-4dbf-874a-fcbd85670d2e","Type":"ContainerDied","Data":"c56c93bf06d8401a57f26577a20c582f266e0335c53ec931a23f8a962d837b7c"}
Nov 02 13:02:00.172143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:00.172025    5879 scope.go:122] "RemoveContainer" containerID="be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125"
Nov 02 13:02:00.172143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:00.171951    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"
Nov 02 13:02:00.185542 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:00.185517    5879 scope.go:122] "RemoveContainer" containerID="be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125"
Nov 02 13:02:00.185879 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:02:00.185834    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125\": container with ID starting with be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125 not found: ID does not exist" containerID="be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125"
Nov 02 13:02:00.185879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:00.185864    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125"} err="failed to get container status \"be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125\": rpc error: code = NotFound desc = could not find container \"be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125\": container with ID starting with be956ab986e343a84722ac262ec50800c3c4ba0aea34efece93ccec3bcf92125 not found: ID does not exist"
Nov 02 13:02:00.220258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:00.220223    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:00.229443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:00.229417    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"]
Nov 02 13:02:00.242761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:00.242736    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-ecec44/autores-model-ecec44-kserve-8645bc99d9-czhjd"]
Nov 02 13:02:00.354720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:00.354627    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:00.742080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:00.742040    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:01.199810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:01.199776    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:01.951313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:01.951280    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:02.030536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:02.023356    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce2991e-3460-4dbf-874a-fcbd85670d2e" path="/var/lib/kubelet/pods/cce2991e-3460-4dbf-874a-fcbd85670d2e/volumes"
Nov 02 13:02:02.645722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:02.645685    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:03.743319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:03.743282    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:03.874460 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:03.874425    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:03.960382 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:03.960349    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"]
Nov 02 13:02:04.019955 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:04.019873    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:04.160533 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:04.160490    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:04.387123 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:04.387094    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:04.961068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:04.961036    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:05.114251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.114213    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:05.780261 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.780226    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:05.868206 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.868167    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"]
Nov 02 13:02:05.868863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.868847    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cce2991e-3460-4dbf-874a-fcbd85670d2e" containerName="main"
Nov 02 13:02:05.868927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.868866    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce2991e-3460-4dbf-874a-fcbd85670d2e" containerName="main"
Nov 02 13:02:05.869040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.869027    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce2991e-3460-4dbf-874a-fcbd85670d2e" containerName="main"
Nov 02 13:02:05.874468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.874442    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"
Nov 02 13:02:05.894653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.894620    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"]
Nov 02 13:02:05.896619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.896591    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-90cda9\""
Nov 02 13:02:05.929359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.929326    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:05.939141 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.939106    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"]
Nov 02 13:02:05.975364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.975326    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"
Nov 02 13:02:05.979717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:05.978329    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"]
Nov 02 13:02:06.045080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.044211    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-90cda9-dockercfg-ntbbb\""
Nov 02 13:02:06.098306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.098266    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efdd27e2-57d3-458f-8f9b-ec45183a701d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4\" (UID: \"efdd27e2-57d3-458f-8f9b-ec45183a701d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"
Nov 02 13:02:06.098522 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.098485    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4r9m\" (UniqueName: \"kubernetes.io/projected/efdd27e2-57d3-458f-8f9b-ec45183a701d-kube-api-access-p4r9m\") pod \"payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4\" (UID: \"efdd27e2-57d3-458f-8f9b-ec45183a701d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"
Nov 02 13:02:06.098600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.098523    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glrwj\" (UniqueName: \"kubernetes.io/projected/c51f6132-b50e-4767-b506-45e1e3a642b5-kube-api-access-glrwj\") pod \"payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk\" (UID: \"c51f6132-b50e-4767-b506-45e1e3a642b5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"
Nov 02 13:02:06.098674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.098618    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51f6132-b50e-4767-b506-45e1e3a642b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk\" (UID: \"c51f6132-b50e-4767-b506-45e1e3a642b5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"
Nov 02 13:02:06.110912 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.110880    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:06.200536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.200504    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4r9m\" (UniqueName: \"kubernetes.io/projected/efdd27e2-57d3-458f-8f9b-ec45183a701d-kube-api-access-p4r9m\") pod \"payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4\" (UID: \"efdd27e2-57d3-458f-8f9b-ec45183a701d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"
Nov 02 13:02:06.200536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.200543    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glrwj\" (UniqueName: \"kubernetes.io/projected/c51f6132-b50e-4767-b506-45e1e3a642b5-kube-api-access-glrwj\") pod \"payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk\" (UID: \"c51f6132-b50e-4767-b506-45e1e3a642b5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"
Nov 02 13:02:06.200835 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.200622    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51f6132-b50e-4767-b506-45e1e3a642b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk\" (UID: \"c51f6132-b50e-4767-b506-45e1e3a642b5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"
Nov 02 13:02:06.200835 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.200709    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efdd27e2-57d3-458f-8f9b-ec45183a701d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4\" (UID: \"efdd27e2-57d3-458f-8f9b-ec45183a701d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"
Nov 02 13:02:06.201672 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.201643    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51f6132-b50e-4767-b506-45e1e3a642b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk\" (UID: \"c51f6132-b50e-4767-b506-45e1e3a642b5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"
Nov 02 13:02:06.202057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.202026    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efdd27e2-57d3-458f-8f9b-ec45183a701d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4\" (UID: \"efdd27e2-57d3-458f-8f9b-ec45183a701d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"
Nov 02 13:02:06.228349 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.228317    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4r9m\" (UniqueName: \"kubernetes.io/projected/efdd27e2-57d3-458f-8f9b-ec45183a701d-kube-api-access-p4r9m\") pod \"payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4\" (UID: \"efdd27e2-57d3-458f-8f9b-ec45183a701d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"
Nov 02 13:02:06.228499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.228321    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glrwj\" (UniqueName: \"kubernetes.io/projected/c51f6132-b50e-4767-b506-45e1e3a642b5-kube-api-access-glrwj\") pod \"payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk\" (UID: \"c51f6132-b50e-4767-b506-45e1e3a642b5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"
Nov 02 13:02:06.404367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.404337    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"
Nov 02 13:02:06.493945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.493910    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"
Nov 02 13:02:06.709509 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:02:06.709461    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51f6132_b50e_4767_b506_45e1e3a642b5.slice/crio-4205a6ddc8ae16ddeac7a554c60953e6237ea9c500c29aa2747d09f4931076cb WatchSource:0}: Error finding container 4205a6ddc8ae16ddeac7a554c60953e6237ea9c500c29aa2747d09f4931076cb: Status 404 returned error can't find the container with id 4205a6ddc8ae16ddeac7a554c60953e6237ea9c500c29aa2747d09f4931076cb
Nov 02 13:02:06.710450 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.710427    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"]
Nov 02 13:02:06.821743 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:02:06.821714    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdd27e2_57d3_458f_8f9b_ec45183a701d.slice/crio-28e2263c2c9e621b6cb4a7d8cd5d110c665153cc046f759eb3077c328900c817 WatchSource:0}: Error finding container 28e2263c2c9e621b6cb4a7d8cd5d110c665153cc046f759eb3077c328900c817: Status 404 returned error can't find the container with id 28e2263c2c9e621b6cb4a7d8cd5d110c665153cc046f759eb3077c328900c817
Nov 02 13:02:06.822241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:06.822220    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"]
Nov 02 13:02:07.032822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:07.032733    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"]
Nov 02 13:02:07.035708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:07.035686    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ae37406-kxmx2"]
Nov 02 13:02:07.233327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:07.233285    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk" event={"ID":"c51f6132-b50e-4767-b506-45e1e3a642b5","Type":"ContainerStarted","Data":"d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff"}
Nov 02 13:02:07.233545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:07.233363    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk" event={"ID":"c51f6132-b50e-4767-b506-45e1e3a642b5","Type":"ContainerStarted","Data":"4205a6ddc8ae16ddeac7a554c60953e6237ea9c500c29aa2747d09f4931076cb"}
Nov 02 13:02:07.233545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:07.233393    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"
Nov 02 13:02:07.235170 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:07.235140    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4" event={"ID":"efdd27e2-57d3-458f-8f9b-ec45183a701d","Type":"ContainerStarted","Data":"dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef"}
Nov 02 13:02:07.235298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:07.235201    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"
Nov 02 13:02:07.235298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:07.235230    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4" event={"ID":"efdd27e2-57d3-458f-8f9b-ec45183a701d","Type":"ContainerStarted","Data":"28e2263c2c9e621b6cb4a7d8cd5d110c665153cc046f759eb3077c328900c817"}
Nov 02 13:02:07.281748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:07.281693    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk" podStartSLOduration=2.281679133 podStartE2EDuration="2.281679133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:07.279558469 +0000 UTC m=+2073.386261320" watchObservedRunningTime="2026-11-02 13:02:07.281679133 +0000 UTC m=+2073.388381984"
Nov 02 13:02:07.325466 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:07.325344    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4" podStartSLOduration=2.32533113 podStartE2EDuration="2.32533113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:07.323253146 +0000 UTC m=+2073.429955996" watchObservedRunningTime="2026-11-02 13:02:07.32533113 +0000 UTC m=+2073.432033972"
Nov 02 13:02:07.788398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:07.788356    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:08.018114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:08.018078    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d4fdad1-9361-4f4e-8f4d-d68dae46d602" path="/var/lib/kubelet/pods/9d4fdad1-9361-4f4e-8f4d-d68dae46d602/volumes"
Nov 02 13:02:08.159294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:08.159268    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"
Nov 02 13:02:08.159699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:08.159330    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"
Nov 02 13:02:08.876731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:08.876700    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:08.961032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:08.960980    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb" podUID="9bc4e19c-58ab-4a52-8496-c1aea5c588ae" containerName="payload-pre-processing" containerID="cri-o://1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43" gracePeriod=25
Nov 02 13:02:09.031049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.030984    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:09.247582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.247556    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"
Nov 02 13:02:09.252328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.252299    5879 generic.go:356] "Generic (PLEG): container finished" podID="9bc4e19c-58ab-4a52-8496-c1aea5c588ae" containerID="1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43" exitCode=0
Nov 02 13:02:09.252480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.252339    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"
Nov 02 13:02:09.252480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.252377    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb" event={"ID":"9bc4e19c-58ab-4a52-8496-c1aea5c588ae","Type":"ContainerDied","Data":"1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43"}
Nov 02 13:02:09.252480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.252431    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb" event={"ID":"9bc4e19c-58ab-4a52-8496-c1aea5c588ae","Type":"ContainerDied","Data":"f523aadb9cee467cd3c0f1bcda6ca80910e13a3e266992fd2a46432407b3af37"}
Nov 02 13:02:09.252480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.252457    5879 scope.go:122] "RemoveContainer" containerID="1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43"
Nov 02 13:02:09.264599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.264576    5879 scope.go:122] "RemoveContainer" containerID="1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43"
Nov 02 13:02:09.265083 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:02:09.265045    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43\": container with ID starting with 1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43 not found: ID does not exist" containerID="1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43"
Nov 02 13:02:09.265197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.265086    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43"} err="failed to get container status \"1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43\": rpc error: code = NotFound desc = could not find container \"1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43\": container with ID starting with 1f63ce5e5080228d54f4dc51158477c93e67e210b05bc994d88eb6d7f3887a43 not found: ID does not exist"
Nov 02 13:02:09.345302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.345264    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-kube-api-access-m2j85\" (UniqueName: \"kubernetes.io/projected/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-kube-api-access-m2j85\") pod \"9bc4e19c-58ab-4a52-8496-c1aea5c588ae\" (UID: \"9bc4e19c-58ab-4a52-8496-c1aea5c588ae\") "
Nov 02 13:02:09.345524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.345367    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-plugins-config-volume\") pod \"9bc4e19c-58ab-4a52-8496-c1aea5c588ae\" (UID: \"9bc4e19c-58ab-4a52-8496-c1aea5c588ae\") "
Nov 02 13:02:09.345820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.345794    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-plugins-config-volume" pod "9bc4e19c-58ab-4a52-8496-c1aea5c588ae" (UID: "9bc4e19c-58ab-4a52-8496-c1aea5c588ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:02:09.347697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.347675    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-kube-api-access-m2j85" pod "9bc4e19c-58ab-4a52-8496-c1aea5c588ae" (UID: "9bc4e19c-58ab-4a52-8496-c1aea5c588ae"). InnerVolumeSpecName "kube-api-access-m2j85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:02:09.447646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.447556    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2j85\" (UniqueName: \"kubernetes.io/projected/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-kube-api-access-m2j85\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:02:09.447646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.447586    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc4e19c-58ab-4a52-8496-c1aea5c588ae-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:02:09.610486 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.610448    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"]
Nov 02 13:02:09.614268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.614235    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb"]
Nov 02 13:02:09.627740 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.627717    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:09.762277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.762187    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:09.888565 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:09.888531    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:10.017807 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.017724    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc4e19c-58ab-4a52-8496-c1aea5c588ae" path="/var/lib/kubelet/pods/9bc4e19c-58ab-4a52-8496-c1aea5c588ae/volumes"
Nov 02 13:02:10.157855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.157814    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-autores-ecec44-5bc4c88cf-fnghb" podUID="9bc4e19c-58ab-4a52-8496-c1aea5c588ae" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.187:50052\" within 1s: context deadline exceeded"
Nov 02 13:02:10.158088 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:02:10.157898    5879 logging.go:55] [core] [Channel #1334 SubChannel #1335]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.187:50052", ServerName: "10.134.0.187:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.187:50052: operation was canceled"
Nov 02 13:02:10.747808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.747769    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:10.866599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.866564    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"]
Nov 02 13:02:10.867261 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.867245    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9bc4e19c-58ab-4a52-8496-c1aea5c588ae" containerName="payload-pre-processing"
Nov 02 13:02:10.867326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.867263    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc4e19c-58ab-4a52-8496-c1aea5c588ae" containerName="payload-pre-processing"
Nov 02 13:02:10.867407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.867397    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc4e19c-58ab-4a52-8496-c1aea5c588ae" containerName="payload-pre-processing"
Nov 02 13:02:10.872270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.872251    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"
Nov 02 13:02:10.888555 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.888528    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:10.891799 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.891777    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ef5efe-dockercfg-trrj7\""
Nov 02 13:02:10.906023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.905538    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ef5efe\""
Nov 02 13:02:10.911319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.911286    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"]
Nov 02 13:02:10.968557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.968511    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4vp\" (UniqueName: \"kubernetes.io/projected/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-kube-api-access-mp4vp\") pod \"payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw\" (UID: \"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"
Nov 02 13:02:10.968831 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:10.968627    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw\" (UID: \"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"
Nov 02 13:02:11.027068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.021681    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"]
Nov 02 13:02:11.032018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.029617    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"
Nov 02 13:02:11.035268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.035231    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"]
Nov 02 13:02:11.072536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.070428    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4vp\" (UniqueName: \"kubernetes.io/projected/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-kube-api-access-mp4vp\") pod \"payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw\" (UID: \"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"
Nov 02 13:02:11.074923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.074846    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw\" (UID: \"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"
Nov 02 13:02:11.074923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.074911    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwk92\" (UniqueName: \"kubernetes.io/projected/164d08f2-4367-4d53-a393-1995d9a4d12e-kube-api-access-vwk92\") pod \"payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7\" (UID: \"164d08f2-4367-4d53-a393-1995d9a4d12e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"
Nov 02 13:02:11.075765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.075735    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164d08f2-4367-4d53-a393-1995d9a4d12e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7\" (UID: \"164d08f2-4367-4d53-a393-1995d9a4d12e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"
Nov 02 13:02:11.077096 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.077047    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw\" (UID: \"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"
Nov 02 13:02:11.100240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.100209    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4vp\" (UniqueName: \"kubernetes.io/projected/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-kube-api-access-mp4vp\") pod \"payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw\" (UID: \"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"
Nov 02 13:02:11.176585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.176553    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwk92\" (UniqueName: \"kubernetes.io/projected/164d08f2-4367-4d53-a393-1995d9a4d12e-kube-api-access-vwk92\") pod \"payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7\" (UID: \"164d08f2-4367-4d53-a393-1995d9a4d12e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"
Nov 02 13:02:11.176798 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.176702    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164d08f2-4367-4d53-a393-1995d9a4d12e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7\" (UID: \"164d08f2-4367-4d53-a393-1995d9a4d12e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"
Nov 02 13:02:11.177632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.177609    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164d08f2-4367-4d53-a393-1995d9a4d12e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7\" (UID: \"164d08f2-4367-4d53-a393-1995d9a4d12e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"
Nov 02 13:02:11.186328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.186306    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"
Nov 02 13:02:11.201269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.201242    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwk92\" (UniqueName: \"kubernetes.io/projected/164d08f2-4367-4d53-a393-1995d9a4d12e-kube-api-access-vwk92\") pod \"payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7\" (UID: \"164d08f2-4367-4d53-a393-1995d9a4d12e\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"
Nov 02 13:02:11.355947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.355857    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"
Nov 02 13:02:11.501046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.500717    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"]
Nov 02 13:02:11.501046 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:02:11.500719    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed43f8f_48f7_46a7_8fc9_f96f0a13d1e2.slice/crio-0451713003ee86f06885bbba16c389939c50a829826d1d795348247bef376da8 WatchSource:0}: Error finding container 0451713003ee86f06885bbba16c389939c50a829826d1d795348247bef376da8: Status 404 returned error can't find the container with id 0451713003ee86f06885bbba16c389939c50a829826d1d795348247bef376da8
Nov 02 13:02:11.508259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.508229    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:11.674317 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:02:11.674278    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164d08f2_4367_4d53_a393_1995d9a4d12e.slice/crio-64337e1ac6dcc5f0cf3d14f082987c3e42eba5f1a60427a779a9101d3fb5ebde WatchSource:0}: Error finding container 64337e1ac6dcc5f0cf3d14f082987c3e42eba5f1a60427a779a9101d3fb5ebde: Status 404 returned error can't find the container with id 64337e1ac6dcc5f0cf3d14f082987c3e42eba5f1a60427a779a9101d3fb5ebde
Nov 02 13:02:11.674976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.674955    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"]
Nov 02 13:02:11.917366 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:11.917311    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:12.289103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:12.289054    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw" event={"ID":"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2","Type":"ContainerStarted","Data":"e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec"}
Nov 02 13:02:12.289335 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:12.289153    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"
Nov 02 13:02:12.289335 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:12.289178    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw" event={"ID":"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2","Type":"ContainerStarted","Data":"0451713003ee86f06885bbba16c389939c50a829826d1d795348247bef376da8"}
Nov 02 13:02:12.290817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:12.290792    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7" event={"ID":"164d08f2-4367-4d53-a393-1995d9a4d12e","Type":"ContainerStarted","Data":"1bb2c6ddae673aaa209857cc7a3fe45b00234dd40e166c178f894fb8b77bbfa6"}
Nov 02 13:02:12.290958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:12.290832    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7" event={"ID":"164d08f2-4367-4d53-a393-1995d9a4d12e","Type":"ContainerStarted","Data":"64337e1ac6dcc5f0cf3d14f082987c3e42eba5f1a60427a779a9101d3fb5ebde"}
Nov 02 13:02:12.290958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:12.290865    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"
Nov 02 13:02:12.336492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:12.336429    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw" podStartSLOduration=2.336412876 podStartE2EDuration="2.336412876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:12.334267952 +0000 UTC m=+2078.440970803" watchObservedRunningTime="2026-11-02 13:02:12.336412876 +0000 UTC m=+2078.443115728"
Nov 02 13:02:12.381596 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:12.381532    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7" podStartSLOduration=2.381516026 podStartE2EDuration="2.381516026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:12.380923243 +0000 UTC m=+2078.487626094" watchObservedRunningTime="2026-11-02 13:02:12.381516026 +0000 UTC m=+2078.488218878"
Nov 02 13:02:13.387894 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:13.387861    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:14.008967 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:14.008932    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:14.823182 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:02:14.823148    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:02:15.529417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:15.529376    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:17.051916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:17.051879    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:17.771498 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:17.771459    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:17.902380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:17.902349    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:18.248429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:18.248390    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"
Nov 02 13:02:18.248935 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:18.248838    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"
Nov 02 13:02:18.429501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:18.429472    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:18.737359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:18.737317    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:18.853904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:18.853868    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:20.498386 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:20.498352    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:20.810843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:20.810752    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:21.414229 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:21.413818    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:21.549284 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:21.549244    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:22.466372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:22.466338    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:23.307376 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:23.307347    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"
Nov 02 13:02:23.307806 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:23.307414    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"
Nov 02 13:02:24.201483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:24.201454    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:25.138199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:25.138162    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:25.296769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:25.296732    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:26.041286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:26.041256    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:26.152623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:26.152579    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:26.271009 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:26.270966    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:27.240772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.240723    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:27.365405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.365360    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:27.388807 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.388776    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"]
Nov 02 13:02:27.393861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.393832    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"
Nov 02 13:02:27.412790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.412761    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"]
Nov 02 13:02:27.419878 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.419840    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg\" (UID: \"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"
Nov 02 13:02:27.420113 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.420031    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxvcc\" (UniqueName: \"kubernetes.io/projected/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-kube-api-access-wxvcc\") pod \"payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg\" (UID: \"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"
Nov 02 13:02:27.421112 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.421088    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-14fcb6\""
Nov 02 13:02:27.421266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.421244    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-14fcb6-dockercfg-r95pr\""
Nov 02 13:02:27.482499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.482452    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"]
Nov 02 13:02:27.487246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.487218    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"
Nov 02 13:02:27.507392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.507362    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"]
Nov 02 13:02:27.521335 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.521294    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg\" (UID: \"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"
Nov 02 13:02:27.521570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.521428    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxvcc\" (UniqueName: \"kubernetes.io/projected/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-kube-api-access-wxvcc\") pod \"payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg\" (UID: \"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"
Nov 02 13:02:27.521570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.521486    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e974ec08-ae6a-490c-946d-f04c6e9ea509-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n\" (UID: \"e974ec08-ae6a-490c-946d-f04c6e9ea509\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"
Nov 02 13:02:27.521732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.521597    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpkzq\" (UniqueName: \"kubernetes.io/projected/e974ec08-ae6a-490c-946d-f04c6e9ea509-kube-api-access-vpkzq\") pod \"payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n\" (UID: \"e974ec08-ae6a-490c-946d-f04c6e9ea509\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"
Nov 02 13:02:27.522214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.522188    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg\" (UID: \"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"
Nov 02 13:02:27.548076 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.548045    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxvcc\" (UniqueName: \"kubernetes.io/projected/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-kube-api-access-wxvcc\") pod \"payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg\" (UID: \"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"
Nov 02 13:02:27.622938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.622892    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpkzq\" (UniqueName: \"kubernetes.io/projected/e974ec08-ae6a-490c-946d-f04c6e9ea509-kube-api-access-vpkzq\") pod \"payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n\" (UID: \"e974ec08-ae6a-490c-946d-f04c6e9ea509\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"
Nov 02 13:02:27.623225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.623202    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e974ec08-ae6a-490c-946d-f04c6e9ea509-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n\" (UID: \"e974ec08-ae6a-490c-946d-f04c6e9ea509\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"
Nov 02 13:02:27.624284 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.624253    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e974ec08-ae6a-490c-946d-f04c6e9ea509-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n\" (UID: \"e974ec08-ae6a-490c-946d-f04c6e9ea509\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"
Nov 02 13:02:27.649012 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.648970    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpkzq\" (UniqueName: \"kubernetes.io/projected/e974ec08-ae6a-490c-946d-f04c6e9ea509-kube-api-access-vpkzq\") pod \"payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n\" (UID: \"e974ec08-ae6a-490c-946d-f04c6e9ea509\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"
Nov 02 13:02:27.711413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.711367    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"
Nov 02 13:02:27.803791 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:27.803508    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"
Nov 02 13:02:28.021765 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:02:28.021736    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee045bd7_66e5_4bd3_97c9_1c0e1e7c8659.slice/crio-5f61f491c9d84be17b714715aa932bebb5772e6e27f6e68e9b983b678a71f90d WatchSource:0}: Error finding container 5f61f491c9d84be17b714715aa932bebb5772e6e27f6e68e9b983b678a71f90d: Status 404 returned error can't find the container with id 5f61f491c9d84be17b714715aa932bebb5772e6e27f6e68e9b983b678a71f90d
Nov 02 13:02:28.022299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:28.022275    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"]
Nov 02 13:02:28.125687 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:02:28.125655    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode974ec08_ae6a_490c_946d_f04c6e9ea509.slice/crio-e0460e4b82766c8dd858ab46d88511b6f526aee641147b00fe24f7a27e04f1d9 WatchSource:0}: Error finding container e0460e4b82766c8dd858ab46d88511b6f526aee641147b00fe24f7a27e04f1d9: Status 404 returned error can't find the container with id e0460e4b82766c8dd858ab46d88511b6f526aee641147b00fe24f7a27e04f1d9
Nov 02 13:02:28.129020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:28.128982    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"]
Nov 02 13:02:28.130320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:28.130296    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:28.424259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:28.424221    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg" event={"ID":"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659","Type":"ContainerStarted","Data":"08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e"}
Nov 02 13:02:28.424707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:28.424283    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg" event={"ID":"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659","Type":"ContainerStarted","Data":"5f61f491c9d84be17b714715aa932bebb5772e6e27f6e68e9b983b678a71f90d"}
Nov 02 13:02:28.424707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:28.424309    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"
Nov 02 13:02:28.426139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:28.426109    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n" event={"ID":"e974ec08-ae6a-490c-946d-f04c6e9ea509","Type":"ContainerStarted","Data":"2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb"}
Nov 02 13:02:28.426275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:28.426157    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n" event={"ID":"e974ec08-ae6a-490c-946d-f04c6e9ea509","Type":"ContainerStarted","Data":"e0460e4b82766c8dd858ab46d88511b6f526aee641147b00fe24f7a27e04f1d9"}
Nov 02 13:02:28.426275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:28.426182    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"
Nov 02 13:02:28.468013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:28.467950    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg" podStartSLOduration=1.467936551 podStartE2EDuration="1.467936551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:28.466743872 +0000 UTC m=+2094.573446722" watchObservedRunningTime="2026-11-02 13:02:28.467936551 +0000 UTC m=+2094.574639401"
Nov 02 13:02:28.510628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:28.510570    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n" podStartSLOduration=1.5105562049999999 podStartE2EDuration="1.510556205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:28.509980796 +0000 UTC m=+2094.616683641" watchObservedRunningTime="2026-11-02 13:02:28.510556205 +0000 UTC m=+2094.617259055"
Nov 02 13:02:29.680562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:29.680521    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:31.254926 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:31.254883    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:32.828269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:32.828233    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:34.449970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:34.449928    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:36.113332 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:36.113290    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:37.446461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:37.446429    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:37.633318 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:37.633240    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:37.753674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:37.753543    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:39.403626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:39.403590    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:39.439587 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:39.438311    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"
Nov 02 13:02:39.441057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:39.440120    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"
Nov 02 13:02:41.011450 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:41.011417    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:42.638218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:42.638178    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:42.816456 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:42.816420    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:44.356703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:44.356665    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:44.822406 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:02:44.822369    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:02:46.027952 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:46.027923    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:47.568197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:47.568157    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:48.676596 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:48.676560    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:49.232243 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:49.232210    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:49.785098 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:49.785065    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:49.920104 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:49.920068    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:50.090737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:50.090625    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:50.401250 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:50.401215    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:50.611549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:50.611511    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:50.738195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:50.738115    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:51.063270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:51.063183    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:51.190249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:51.190213    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:52.701980 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.701933    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:52.840121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.840010    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:52.938204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.938166    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"]
Nov 02 13:02:52.946063 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.946034    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:52.961642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.961607    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"]
Nov 02 13:02:52.970519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.970487    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"agc-cred-48094dd3d1ec\""
Nov 02 13:02:52.970924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.970902    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"kube-root-ca.crt\""
Nov 02 13:02:52.971087 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.970936    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"openshift-service-ca.crt\""
Nov 02 13:02:52.971213 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.971023    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"routing-overlay\""
Nov 02 13:02:52.976197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.976169    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"payload-processing-external-model-plugins-e2e-extmod-b-00d004\""
Nov 02 13:02:52.976346 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.976217    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"payload-processing-external-model-e2e--8054d3fe-dockercfg-fgrk9\""
Nov 02 13:02:52.984254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.984230    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:52.984355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.984310    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:52.984451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.984423    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgstd\" (UniqueName: \"kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-kube-api-access-sgstd\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:52.984609 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:52.984594    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:53.086128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:53.086076    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:53.115046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:53.087426    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:53.117720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:53.117682    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:53.120978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:53.118630    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:53.121305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:53.121280    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:53.121858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:53.121838    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgstd\" (UniqueName: \"kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-kube-api-access-sgstd\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:53.127596 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:53.126239    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:53.147394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:53.147366    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgstd\" (UniqueName: \"kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-kube-api-access-sgstd\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:53.263930 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:53.263840    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:53.573101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:53.573068    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"]
Nov 02 13:02:53.574298 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:02:53.574272    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcd35dc_690c_4034_a5de_f4ce472c9f22.slice/crio-ca258cf5f384270498586ce7c4f7dbad72125fbe82cdcff65c17a2400a7cca7c WatchSource:0}: Error finding container ca258cf5f384270498586ce7c4f7dbad72125fbe82cdcff65c17a2400a7cca7c: Status 404 returned error can't find the container with id ca258cf5f384270498586ce7c4f7dbad72125fbe82cdcff65c17a2400a7cca7c
Nov 02 13:02:53.634347 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:53.634314    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c" event={"ID":"0bcd35dc-690c-4034-a5de-f4ce472c9f22","Type":"ContainerStarted","Data":"ca258cf5f384270498586ce7c4f7dbad72125fbe82cdcff65c17a2400a7cca7c"}
Nov 02 13:02:54.328900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:54.328861    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:54.643512 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:54.643474    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c" event={"ID":"0bcd35dc-690c-4034-a5de-f4ce472c9f22","Type":"ContainerStarted","Data":"b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c"}
Nov 02 13:02:54.643708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:54.643678    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:02:54.689300 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:54.689243    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c" podStartSLOduration=2.68922953 podStartE2EDuration="2.68922953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:02:54.687063046 +0000 UTC m=+2120.793765896" watchObservedRunningTime="2026-11-02 13:02:54.68922953 +0000 UTC m=+2120.795932463"
Nov 02 13:02:55.933296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:55.933262    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:57.538195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:57.538149    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:59.032739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:59.032699    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:59.156900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:59.156869    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:02:59.272428 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:02:59.272394    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:00.730804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:00.730765    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:02.348124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:02.348087    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:03.912311 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:03.912279    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:05.327581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:05.327542    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:05.654412 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:05.654384    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:03:05.767135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:05.767105    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:06.481917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:06.481883    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:06.682389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:06.682350    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:06.817687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:06.817612    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:07.108335 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:07.108243    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:07.248369 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:07.248338    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:07.380020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:07.379969    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:07.752134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:07.752059    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:07.924397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:07.924360    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:08.045766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:08.045688    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:09.339464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.339272    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:09.539130 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.539081    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:09.638488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.638458    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"]
Nov 02 13:03:09.644212 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.644190    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.675015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.674499    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e422b\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0e422b\""
Nov 02 13:03:09.675015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.674659    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e422b\"/\"agc-cred-138df3b93cd6\""
Nov 02 13:03:09.675015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.674722    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e422b\"/\"openshift-service-ca.crt\""
Nov 02 13:03:09.675015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.674844    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0e422b\"/\"payload-processing-external-model-e2e--66c6adfb-dockercfg-pd9tb\""
Nov 02 13:03:09.675015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.674866    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e422b\"/\"routing-overlay\""
Nov 02 13:03:09.675015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.674975    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0e422b\"/\"kube-root-ca.crt\""
Nov 02 13:03:09.678735 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.678714    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"]
Nov 02 13:03:09.764283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.764245    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpm97\" (UniqueName: \"kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-kube-api-access-hpm97\") pod \"payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") " pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.764478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.764313    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") " pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.764478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.764463    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") " pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.764561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.764530    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") " pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.865655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.865610    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") " pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.865933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.865737    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") " pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.865933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.865804    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") " pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.866109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.865982    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpm97\" (UniqueName: \"kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-kube-api-access-hpm97\") pod \"payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") " pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.866661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.866587    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") " pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.866661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.866590    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") " pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.868721 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.868697    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") " pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.921150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.921016    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpm97\" (UniqueName: \"kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-kube-api-access-hpm97\") pod \"payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") " pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:09.961311 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:09.961272    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:10.278086 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:03:10.278054    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0cc0d39_8a32_4852_9267_c4b3926b5685.slice/crio-11d80203f067ebe22e77d018f06213af85406ccfe4203f79ba93380dffd4de0a WatchSource:0}: Error finding container 11d80203f067ebe22e77d018f06213af85406ccfe4203f79ba93380dffd4de0a: Status 404 returned error can't find the container with id 11d80203f067ebe22e77d018f06213af85406ccfe4203f79ba93380dffd4de0a
Nov 02 13:03:10.278210 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:10.278133    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"]
Nov 02 13:03:10.796717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:10.796679    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8" event={"ID":"b0cc0d39-8a32-4852-9267-c4b3926b5685","Type":"ContainerStarted","Data":"11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855"}
Nov 02 13:03:10.797154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:10.796740    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8" event={"ID":"b0cc0d39-8a32-4852-9267-c4b3926b5685","Type":"ContainerStarted","Data":"11d80203f067ebe22e77d018f06213af85406ccfe4203f79ba93380dffd4de0a"}
Nov 02 13:03:10.797154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:10.796919    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:10.843416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:10.843361    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8" podStartSLOduration=1.843344873 podStartE2EDuration="1.843344873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:03:10.840116833 +0000 UTC m=+2136.946819681" watchObservedRunningTime="2026-11-02 13:03:10.843344873 +0000 UTC m=+2136.950047724"
Nov 02 13:03:10.947410 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:10.947352    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:12.578849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:12.578796    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:14.346465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:14.346431    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:14.823135 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:03:14.823102    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:03:16.027122 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:16.027086    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:17.843682 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:17.843641    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:19.524170 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:19.524138    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:20.541267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:20.541233    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:21.130459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:21.130425    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:21.721400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:21.721367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:21.807099 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:21.807054    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:03:21.919458 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:21.919428    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:22.062606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:22.062508    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:22.343564 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:22.343458    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:22.493210 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:22.493176    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:22.626928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:22.626899    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:22.890518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:22.890436    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:23.118519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:23.118489    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:23.243668 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:23.243577    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:24.592248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:24.592217    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:24.794007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:24.793953    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:25.004841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.004806    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"]
Nov 02 13:03:25.009949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.009921    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.028838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.028804    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"]
Nov 02 13:03:25.038798 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.038772    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"agc-cred-8f1d63a606c1\""
Nov 02 13:03:25.039083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.039054    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"kube-root-ca.crt\""
Nov 02 13:03:25.039182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.039161    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"openshift-service-ca.crt\""
Nov 02 13:03:25.039527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.039507    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"payload-processing-external-model-e2e--4b89980e-dockercfg-4g29z\""
Nov 02 13:03:25.042833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.042810    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-90cda9\""
Nov 02 13:03:25.042833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.042812    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"routing-overlay\""
Nov 02 13:03:25.104725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.104689    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4v9g\" (UniqueName: \"kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-kube-api-access-w4v9g\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.104906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.104734    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.104906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.104800    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.105065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.105049    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.220295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.220243    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.220534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.220357    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4v9g\" (UniqueName: \"kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-kube-api-access-w4v9g\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.220534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.220403    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.220534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.220452    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.221598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.221569    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.228543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.227663    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.228543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.228528    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.254274 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.254208    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4v9g\" (UniqueName: \"kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-kube-api-access-w4v9g\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.325722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.325629    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.469973 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.469941    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:25.635818 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:03:25.635787    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5b9b5f_2237_46aa_b406_88f551af2789.slice/crio-f1c28759c8c5728291ee78adcd839e0a4300b28719f80d32302719ac6fddfd3b WatchSource:0}: Error finding container f1c28759c8c5728291ee78adcd839e0a4300b28719f80d32302719ac6fddfd3b: Status 404 returned error can't find the container with id f1c28759c8c5728291ee78adcd839e0a4300b28719f80d32302719ac6fddfd3b
Nov 02 13:03:25.636638 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.636605    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"]
Nov 02 13:03:25.926605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.926509    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp" event={"ID":"2a5b9b5f-2237-46aa-b406-88f551af2789","Type":"ContainerStarted","Data":"22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29"}
Nov 02 13:03:25.926605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.926573    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp" event={"ID":"2a5b9b5f-2237-46aa-b406-88f551af2789","Type":"ContainerStarted","Data":"f1c28759c8c5728291ee78adcd839e0a4300b28719f80d32302719ac6fddfd3b"}
Nov 02 13:03:25.926605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.926606    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:25.970109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:25.970037    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp" podStartSLOduration=1.970019851 podStartE2EDuration="1.970019851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:03:25.969522639 +0000 UTC m=+2152.076225489" watchObservedRunningTime="2026-11-02 13:03:25.970019851 +0000 UTC m=+2152.076722693"
Nov 02 13:03:26.258206 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:26.258108    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:26.819310 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:26.819278    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:27.322263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:27.322228    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:27.567905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:27.567866    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:27.857629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:27.857600    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:29.458160 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:29.458122    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:30.970803 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:30.970770    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:31.101441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:31.101399    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:32.123523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:32.123487    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"]
Nov 02 13:03:32.124067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:32.123709    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" podUID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" containerName="istio-proxy" containerID="cri-o://5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef" gracePeriod=30
Nov 02 13:03:32.212560 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:32.212527    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:32.768466 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:32.768431    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:32.891121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:32.891087    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:33.005958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.005929    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"]
Nov 02 13:03:33.010883 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.010854    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"
Nov 02 13:03:33.027273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.027240    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:33.029476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.029451    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-936529-dockercfg-zcd88\""
Nov 02 13:03:33.029626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.029597    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-936529\""
Nov 02 13:03:33.029698 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.029620    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"]
Nov 02 13:03:33.099695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.099659    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"]
Nov 02 13:03:33.105191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.105166    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"
Nov 02 13:03:33.125483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.125448    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"]
Nov 02 13:03:33.133880 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.133022    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r6pb\" (UniqueName: \"kubernetes.io/projected/111b30d0-5386-4fc5-bc65-6108173dede5-kube-api-access-6r6pb\") pod \"payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq\" (UID: \"111b30d0-5386-4fc5-bc65-6108173dede5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"
Nov 02 13:03:33.133880 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.133119    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111b30d0-5386-4fc5-bc65-6108173dede5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq\" (UID: \"111b30d0-5386-4fc5-bc65-6108173dede5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"
Nov 02 13:03:33.234624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.234591    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r6pb\" (UniqueName: \"kubernetes.io/projected/111b30d0-5386-4fc5-bc65-6108173dede5-kube-api-access-6r6pb\") pod \"payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq\" (UID: \"111b30d0-5386-4fc5-bc65-6108173dede5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"
Nov 02 13:03:33.235025 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.234953    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111b30d0-5386-4fc5-bc65-6108173dede5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq\" (UID: \"111b30d0-5386-4fc5-bc65-6108173dede5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"
Nov 02 13:03:33.235187 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.235097    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2117da57-e909-43d4-a471-c0f4b3c0f13d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx\" (UID: \"2117da57-e909-43d4-a471-c0f4b3c0f13d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"
Nov 02 13:03:33.235187 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.235179    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvk5\" (UniqueName: \"kubernetes.io/projected/2117da57-e909-43d4-a471-c0f4b3c0f13d-kube-api-access-6bvk5\") pod \"payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx\" (UID: \"2117da57-e909-43d4-a471-c0f4b3c0f13d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"
Nov 02 13:03:33.235815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.235789    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111b30d0-5386-4fc5-bc65-6108173dede5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq\" (UID: \"111b30d0-5386-4fc5-bc65-6108173dede5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"
Nov 02 13:03:33.260034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.259985    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r6pb\" (UniqueName: \"kubernetes.io/projected/111b30d0-5386-4fc5-bc65-6108173dede5-kube-api-access-6r6pb\") pod \"payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq\" (UID: \"111b30d0-5386-4fc5-bc65-6108173dede5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"
Nov 02 13:03:33.328040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.327933    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"
Nov 02 13:03:33.336488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.336456    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2117da57-e909-43d4-a471-c0f4b3c0f13d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx\" (UID: \"2117da57-e909-43d4-a471-c0f4b3c0f13d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"
Nov 02 13:03:33.336672 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.336549    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvk5\" (UniqueName: \"kubernetes.io/projected/2117da57-e909-43d4-a471-c0f4b3c0f13d-kube-api-access-6bvk5\") pod \"payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx\" (UID: \"2117da57-e909-43d4-a471-c0f4b3c0f13d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"
Nov 02 13:03:33.337413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.337386    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2117da57-e909-43d4-a471-c0f4b3c0f13d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx\" (UID: \"2117da57-e909-43d4-a471-c0f4b3c0f13d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"
Nov 02 13:03:33.361684 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.361657    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvk5\" (UniqueName: \"kubernetes.io/projected/2117da57-e909-43d4-a471-c0f4b3c0f13d-kube-api-access-6bvk5\") pod \"payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx\" (UID: \"2117da57-e909-43d4-a471-c0f4b3c0f13d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"
Nov 02 13:03:33.423032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.422922    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"
Nov 02 13:03:33.645036 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:03:33.644962    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111b30d0_5386_4fc5_bc65_6108173dede5.slice/crio-9487f0e2f89bc6c41603a2517fc26f6b4157ef4dd80d32b185d11f049f59d3bf WatchSource:0}: Error finding container 9487f0e2f89bc6c41603a2517fc26f6b4157ef4dd80d32b185d11f049f59d3bf: Status 404 returned error can't find the container with id 9487f0e2f89bc6c41603a2517fc26f6b4157ef4dd80d32b185d11f049f59d3bf
Nov 02 13:03:33.646488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.646460    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"]
Nov 02 13:03:33.743029 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:03:33.742966    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2117da57_e909_43d4_a471_c0f4b3c0f13d.slice/crio-3e69eb925a7901b39553ca555cda37124f7313292f199c6eb1d7554546c96da6 WatchSource:0}: Error finding container 3e69eb925a7901b39553ca555cda37124f7313292f199c6eb1d7554546c96da6: Status 404 returned error can't find the container with id 3e69eb925a7901b39553ca555cda37124f7313292f199c6eb1d7554546c96da6
Nov 02 13:03:33.744909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.744886    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"]
Nov 02 13:03:33.992211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.992110    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx" event={"ID":"2117da57-e909-43d4-a471-c0f4b3c0f13d","Type":"ContainerStarted","Data":"a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341"}
Nov 02 13:03:33.992211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.992178    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"
Nov 02 13:03:33.992211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.992206    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx" event={"ID":"2117da57-e909-43d4-a471-c0f4b3c0f13d","Type":"ContainerStarted","Data":"3e69eb925a7901b39553ca555cda37124f7313292f199c6eb1d7554546c96da6"}
Nov 02 13:03:33.993911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.993884    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq" event={"ID":"111b30d0-5386-4fc5-bc65-6108173dede5","Type":"ContainerStarted","Data":"3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca"}
Nov 02 13:03:33.994063 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.993936    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq" event={"ID":"111b30d0-5386-4fc5-bc65-6108173dede5","Type":"ContainerStarted","Data":"9487f0e2f89bc6c41603a2517fc26f6b4157ef4dd80d32b185d11f049f59d3bf"}
Nov 02 13:03:33.994063 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:33.993967    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"
Nov 02 13:03:34.037661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:34.037598    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx" podStartSLOduration=1.037578214 podStartE2EDuration="1.037578214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:03:34.034749565 +0000 UTC m=+2160.141452416" watchObservedRunningTime="2026-11-02 13:03:34.037578214 +0000 UTC m=+2160.144281067"
Nov 02 13:03:34.099954 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:34.099896    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq" podStartSLOduration=2.099879808 podStartE2EDuration="2.099879808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:03:34.098253062 +0000 UTC m=+2160.204955913" watchObservedRunningTime="2026-11-02 13:03:34.099879808 +0000 UTC m=+2160.206582659"
Nov 02 13:03:34.355946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:34.355850    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:35.091616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:35.091582    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"]
Nov 02 13:03:35.091796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:35.091777    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67" podUID="3e00db56-210d-4523-bc46-779067bac1b5" containerName="istio-proxy" containerID="cri-o://5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475" gracePeriod=30
Nov 02 13:03:35.944572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:35.944538    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:35.978846 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:35.978802    5879 scope.go:122] "RemoveContainer" containerID="a26f63870921a4512a8e0d1dfc24a795e35a7fa8bca0c24afcba3c9ff6277f0b"
Nov 02 13:03:36.937842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:36.937811    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:03:37.466698 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.466668    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 13:03:37.502191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502099    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-podinfo\") pod \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") "
Nov 02 13:03:37.502191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502159    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-kube-api-access-tftl2\" (UniqueName: \"kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-kube-api-access-tftl2\") pod \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") "
Nov 02 13:03:37.502392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502232    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-certs\") pod \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") "
Nov 02 13:03:37.502392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502296    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-socket\") pod \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") "
Nov 02 13:03:37.502392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502333    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istiod-ca-cert\") pod \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") "
Nov 02 13:03:37.502392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502373    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-envoy\") pod \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") "
Nov 02 13:03:37.502623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502500    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-data\") pod \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") "
Nov 02 13:03:37.502623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502537    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-certs" pod "729df3ad-162a-47b5-9d93-351ec4ef3fd8" (UID: "729df3ad-162a-47b5-9d93-351ec4ef3fd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:37.502623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502554    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-credential-socket\") pod \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") "
Nov 02 13:03:37.502623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502609    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-socket" pod "729df3ad-162a-47b5-9d93-351ec4ef3fd8" (UID: "729df3ad-162a-47b5-9d93-351ec4ef3fd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:37.502859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502712    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-token\" (UniqueName: \"kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-token\") pod \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\" (UID: \"729df3ad-162a-47b5-9d93-351ec4ef3fd8\") "
Nov 02 13:03:37.502859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502833    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-credential-socket" pod "729df3ad-162a-47b5-9d93-351ec4ef3fd8" (UID: "729df3ad-162a-47b5-9d93-351ec4ef3fd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:37.502975 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502880    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-data" pod "729df3ad-162a-47b5-9d93-351ec4ef3fd8" (UID: "729df3ad-162a-47b5-9d93-351ec4ef3fd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:37.502975 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.502901    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istiod-ca-cert" pod "729df3ad-162a-47b5-9d93-351ec4ef3fd8" (UID: "729df3ad-162a-47b5-9d93-351ec4ef3fd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:03:37.503301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.503282    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:37.503381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.503304    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:37.503381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.503317    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:37.503381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.503330    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:37.503381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.503343    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:37.504841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.504813    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-kube-api-access-tftl2" pod "729df3ad-162a-47b5-9d93-351ec4ef3fd8" (UID: "729df3ad-162a-47b5-9d93-351ec4ef3fd8"). InnerVolumeSpecName "kube-api-access-tftl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:03:37.504923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.504898    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-envoy" pod "729df3ad-162a-47b5-9d93-351ec4ef3fd8" (UID: "729df3ad-162a-47b5-9d93-351ec4ef3fd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:37.505246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.505225    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-podinfo" pod "729df3ad-162a-47b5-9d93-351ec4ef3fd8" (UID: "729df3ad-162a-47b5-9d93-351ec4ef3fd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:03:37.505314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.505256    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-token" pod "729df3ad-162a-47b5-9d93-351ec4ef3fd8" (UID: "729df3ad-162a-47b5-9d93-351ec4ef3fd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:03:37.565950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.565909    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:37.605363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.605334    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:37.605363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.605363    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:37.605575 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.605377    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tftl2\" (UniqueName: \"kubernetes.io/projected/729df3ad-162a-47b5-9d93-351ec4ef3fd8-kube-api-access-tftl2\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:37.605575 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.605390    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/729df3ad-162a-47b5-9d93-351ec4ef3fd8-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:37.681177 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.681136    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:37.819365 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.819267    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:03:37.819365 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:37.819335    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67" podUID="3e00db56-210d-4523-bc46-779067bac1b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:03:38.034534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:38.034496    5879 generic.go:356] "Generic (PLEG): container finished" podID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" containerID="5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef" exitCode=0
Nov 02 13:03:38.034707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:38.034560    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"
Nov 02 13:03:38.034707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:38.034570    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" event={"ID":"729df3ad-162a-47b5-9d93-351ec4ef3fd8","Type":"ContainerDied","Data":"5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef"}
Nov 02 13:03:38.034707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:38.034623    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92" event={"ID":"729df3ad-162a-47b5-9d93-351ec4ef3fd8","Type":"ContainerDied","Data":"ceacc54baa00d8583e2c81894b0733b2f9cd513ecdd3a83c994cbacff44e5f6e"}
Nov 02 13:03:38.034707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:38.034644    5879 scope.go:122] "RemoveContainer" containerID="5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef"
Nov 02 13:03:38.046071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:38.046048    5879 scope.go:122] "RemoveContainer" containerID="5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef"
Nov 02 13:03:38.046413 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:03:38.046390    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef\": container with ID starting with 5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef not found: ID does not exist" containerID="5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef"
Nov 02 13:03:38.046491 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:38.046421    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef"} err="failed to get container status \"5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef\": rpc error: code = NotFound desc = could not find container \"5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef\": container with ID starting with 5e2330288578f024ec6940ddabcb9d4dba3e4a64fb278c9f38ec189952336cef not found: ID does not exist"
Nov 02 13:03:38.093899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:38.093807    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"]
Nov 02 13:03:38.108075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:38.108024    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-q7d92"]
Nov 02 13:03:38.928668 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:38.928632    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"]
Nov 02 13:03:38.929112 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:38.928846    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j" podUID="e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" containerName="istio-proxy" containerID="cri-o://b95b5d928133aa7acbb925a30fbf95de474abdf2a9da87282a0566f5ba731d07" gracePeriod=30
Nov 02 13:03:39.159411 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:39.159367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:40.019760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.019728    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" path="/var/lib/kubelet/pods/729df3ad-162a-47b5-9d93-351ec4ef3fd8/volumes"
Nov 02 13:03:40.373954 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.373925    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 13:03:40.445609 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.445572    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-envoy\") pod \"3e00db56-210d-4523-bc46-779067bac1b5\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") "
Nov 02 13:03:40.445871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.445619    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-kube-api-access-nq52j\" (UniqueName: \"kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-kube-api-access-nq52j\") pod \"3e00db56-210d-4523-bc46-779067bac1b5\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") "
Nov 02 13:03:40.445871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.445654    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-certs\") pod \"3e00db56-210d-4523-bc46-779067bac1b5\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") "
Nov 02 13:03:40.445871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.445699    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-socket\") pod \"3e00db56-210d-4523-bc46-779067bac1b5\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") "
Nov 02 13:03:40.445871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.445822    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e00db56-210d-4523-bc46-779067bac1b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e00db56-210d-4523-bc46-779067bac1b5-istio-podinfo\") pod \"3e00db56-210d-4523-bc46-779067bac1b5\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") "
Nov 02 13:03:40.446168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.445926    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-istio-token\" (UniqueName: \"kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-istio-token\") pod \"3e00db56-210d-4523-bc46-779067bac1b5\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") "
Nov 02 13:03:40.446168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.445952    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-certs" pod "3e00db56-210d-4523-bc46-779067bac1b5" (UID: "3e00db56-210d-4523-bc46-779067bac1b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:40.446168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.446117    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-data\") pod \"3e00db56-210d-4523-bc46-779067bac1b5\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") "
Nov 02 13:03:40.446168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.446134    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-socket" pod "3e00db56-210d-4523-bc46-779067bac1b5" (UID: "3e00db56-210d-4523-bc46-779067bac1b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:40.446373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.446180    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-credential-socket\") pod \"3e00db56-210d-4523-bc46-779067bac1b5\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") "
Nov 02 13:03:40.446373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.446218    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e00db56-210d-4523-bc46-779067bac1b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e00db56-210d-4523-bc46-779067bac1b5-istiod-ca-cert\") pod \"3e00db56-210d-4523-bc46-779067bac1b5\" (UID: \"3e00db56-210d-4523-bc46-779067bac1b5\") "
Nov 02 13:03:40.447262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.446853    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:40.447262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.446879    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:40.447262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.447031    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-credential-socket" pod "3e00db56-210d-4523-bc46-779067bac1b5" (UID: "3e00db56-210d-4523-bc46-779067bac1b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:40.447262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.447227    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e00db56-210d-4523-bc46-779067bac1b5-istiod-ca-cert" pod "3e00db56-210d-4523-bc46-779067bac1b5" (UID: "3e00db56-210d-4523-bc46-779067bac1b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:03:40.447262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.447233    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-data" pod "3e00db56-210d-4523-bc46-779067bac1b5" (UID: "3e00db56-210d-4523-bc46-779067bac1b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:40.448459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.448432    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-envoy" pod "3e00db56-210d-4523-bc46-779067bac1b5" (UID: "3e00db56-210d-4523-bc46-779067bac1b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:40.448658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.448634    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e00db56-210d-4523-bc46-779067bac1b5-istio-podinfo" pod "3e00db56-210d-4523-bc46-779067bac1b5" (UID: "3e00db56-210d-4523-bc46-779067bac1b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:03:40.448739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.448724    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-kube-api-access-nq52j" pod "3e00db56-210d-4523-bc46-779067bac1b5" (UID: "3e00db56-210d-4523-bc46-779067bac1b5"). InnerVolumeSpecName "kube-api-access-nq52j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:03:40.448844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.448826    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-istio-token" pod "3e00db56-210d-4523-bc46-779067bac1b5" (UID: "3e00db56-210d-4523-bc46-779067bac1b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:03:40.549050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.548933    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:40.549050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.548965    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq52j\" (UniqueName: \"kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-kube-api-access-nq52j\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:40.549050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.548977    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e00db56-210d-4523-bc46-779067bac1b5-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:40.549050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.549005    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e00db56-210d-4523-bc46-779067bac1b5-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:40.549050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.549018    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:40.549050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.549027    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e00db56-210d-4523-bc46-779067bac1b5-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:40.549050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.549034    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e00db56-210d-4523-bc46-779067bac1b5-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:40.742906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:40.742874    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:41.061635 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:41.061599    5879 generic.go:356] "Generic (PLEG): container finished" podID="3e00db56-210d-4523-bc46-779067bac1b5" containerID="5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475" exitCode=0
Nov 02 13:03:41.062056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:41.061651    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"
Nov 02 13:03:41.062056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:41.061680    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67" event={"ID":"3e00db56-210d-4523-bc46-779067bac1b5","Type":"ContainerDied","Data":"5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475"}
Nov 02 13:03:41.062056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:41.061732    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67" event={"ID":"3e00db56-210d-4523-bc46-779067bac1b5","Type":"ContainerDied","Data":"594d74a5172abff44be200069877ecd7638f15b4a8be8d4d222714504fd88c41"}
Nov 02 13:03:41.062056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:41.061759    5879 scope.go:122] "RemoveContainer" containerID="5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475"
Nov 02 13:03:41.073228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:41.073211    5879 scope.go:122] "RemoveContainer" containerID="5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475"
Nov 02 13:03:41.073527 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:03:41.073505    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475\": container with ID starting with 5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475 not found: ID does not exist" containerID="5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475"
Nov 02 13:03:41.073574 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:41.073535    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475"} err="failed to get container status \"5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475\": rpc error: code = NotFound desc = could not find container \"5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475\": container with ID starting with 5d9d256a8603bdc12d56ccf4d4dff72b5f452c95179d622a435f9b240cf47475 not found: ID does not exist"
Nov 02 13:03:41.120855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:41.120823    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"]
Nov 02 13:03:41.130321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:41.130274    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-vlm67"]
Nov 02 13:03:41.791681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:41.791646    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:42.018787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:42.018747    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e00db56-210d-4523-bc46-779067bac1b5" path="/var/lib/kubelet/pods/3e00db56-210d-4523-bc46-779067bac1b5/volumes"
Nov 02 13:03:42.358405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:42.358373    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:43.163177 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.163138    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:43.554333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.554207    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"]
Nov 02 13:03:43.555010 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.554970    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3e00db56-210d-4523-bc46-779067bac1b5" containerName="istio-proxy"
Nov 02 13:03:43.555010 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.555007    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e00db56-210d-4523-bc46-779067bac1b5" containerName="istio-proxy"
Nov 02 13:03:43.555187 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.555028    5879 cpu_manager.go:414] "containerMap: removing container" podUID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" containerName="istio-proxy"
Nov 02 13:03:43.555187 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.555034    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" containerName="istio-proxy"
Nov 02 13:03:43.555294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.555198    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e00db56-210d-4523-bc46-779067bac1b5" containerName="istio-proxy"
Nov 02 13:03:43.555294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.555208    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="729df3ad-162a-47b5-9d93-351ec4ef3fd8" containerName="istio-proxy"
Nov 02 13:03:43.561207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.561177    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.586710 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.586663    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"]
Nov 02 13:03:43.594266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.594229    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e3396f-openshift-default-dockercfg-9s74s\""
Nov 02 13:03:43.638328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.638296    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:43.691895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.691853    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-podinfo\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.692157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.691974    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-data\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.692157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.692084    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istiod-ca-cert\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.692298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.692186    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-envoy\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.692298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.692232    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-token\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.692298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.692268    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-certs\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.692462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.692380    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txbrq\" (UniqueName: \"kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-kube-api-access-txbrq\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.692462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.692415    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-credential-socket\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.692462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.692431    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-socket\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.799031 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.797805    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:43.803950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.803910    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istiod-ca-cert\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.804224 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.804203    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-envoy\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.804428 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.804380    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-token\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.804568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.804553    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-certs\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.804736 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.804718    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txbrq\" (UniqueName: \"kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-kube-api-access-txbrq\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.804860 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.804845    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-credential-socket\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.804958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.804945    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-socket\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.805149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.805132    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-podinfo\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.805354 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.805338    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-data\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.805938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.805914    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-data\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.806732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.806709    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istiod-ca-cert\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.808675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.807794    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-certs\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.808675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.808070    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-socket\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.808675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.808309    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-credential-socket\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.834221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.834188    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-envoy\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.834698 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.834666    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-podinfo\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.840516 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.840481    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-token\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.841114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.841068    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txbrq\" (UniqueName: \"kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-kube-api-access-txbrq\") pod \"e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") " pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:43.879465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:43.879427    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:44.090160 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.090064    5879 generic.go:356] "Generic (PLEG): container finished" podID="e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" containerID="b95b5d928133aa7acbb925a30fbf95de474abdf2a9da87282a0566f5ba731d07" exitCode=0
Nov 02 13:03:44.090347 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.090147    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j" event={"ID":"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5","Type":"ContainerDied","Data":"b95b5d928133aa7acbb925a30fbf95de474abdf2a9da87282a0566f5ba731d07"}
Nov 02 13:03:44.177633 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.177509    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:44.204138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.204097    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 13:03:44.222039 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:03:44.217423    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d9f2c8_011b_49fa_9f7b_5a36ee074541.slice/crio-3da61e0e02850ef6630e752ffa8bec0ed6edfbfd3d4f92aeb3bdb0c0ff2b8d06 WatchSource:0}: Error finding container 3da61e0e02850ef6630e752ffa8bec0ed6edfbfd3d4f92aeb3bdb0c0ff2b8d06: Status 404 returned error can't find the container with id 3da61e0e02850ef6630e752ffa8bec0ed6edfbfd3d4f92aeb3bdb0c0ff2b8d06
Nov 02 13:03:44.222039 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.219809    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"]
Nov 02 13:03:44.222328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.222056    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:03:44.222328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.222155    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:03:44.222328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.222198    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:03:44.314239 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.314213    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-data\") pod \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") "
Nov 02 13:03:44.314372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.314264    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-certs\") pod \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") "
Nov 02 13:03:44.314372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.314333    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-envoy\") pod \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") "
Nov 02 13:03:44.314461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.314400    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istiod-ca-cert\") pod \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") "
Nov 02 13:03:44.314553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.314527    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-data" pod "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" (UID: "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:44.314601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.314570    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-certs" pod "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" (UID: "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:44.314827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.314807    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istiod-ca-cert" pod "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" (UID: "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:03:44.314894 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.314843    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-token\" (UniqueName: \"kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-token\") pod \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") "
Nov 02 13:03:44.315022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.314980    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-kube-api-access-dj6gj\" (UniqueName: \"kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-kube-api-access-dj6gj\") pod \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") "
Nov 02 13:03:44.315188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.315098    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-podinfo\") pod \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") "
Nov 02 13:03:44.315188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.315151    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-credential-socket\") pod \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") "
Nov 02 13:03:44.315363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.315307    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-socket\") pod \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\" (UID: \"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5\") "
Nov 02 13:03:44.315724 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.315698    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-credential-socket" pod "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" (UID: "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:44.315919 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.315896    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-socket" pod "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" (UID: "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:44.316306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.316285    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:44.316400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.316313    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:44.316400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.316327    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:44.316400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.316343    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:44.316400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.316357    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:44.317205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.317171    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-envoy" pod "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" (UID: "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:03:44.317464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.317435    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-token" pod "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" (UID: "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:03:44.318344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.318311    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-podinfo" pod "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" (UID: "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:03:44.318441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.318368    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-kube-api-access-dj6gj" pod "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" (UID: "e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5"). InnerVolumeSpecName "kube-api-access-dj6gj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:03:44.417838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.417793    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:44.417838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.417838    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:44.417838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.417853    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:44.418279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:44.417868    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj6gj\" (UniqueName: \"kubernetes.io/projected/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5-kube-api-access-dj6gj\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:03:44.823524 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:03:44.823415    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:03:45.005871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.005836    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"
Nov 02 13:03:45.006148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.006128    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"
Nov 02 13:03:45.100907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.100832    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"
Nov 02 13:03:45.101097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.100832    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j" event={"ID":"e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5","Type":"ContainerDied","Data":"44d5fb105098b64fbe2452a428d5da5ebcee2a8e0d4407b772e199e32fea6300"}
Nov 02 13:03:45.101097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.100979    5879 scope.go:122] "RemoveContainer" containerID="b95b5d928133aa7acbb925a30fbf95de474abdf2a9da87282a0566f5ba731d07"
Nov 02 13:03:45.103147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.103118    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" event={"ID":"40d9f2c8-011b-49fa-9f7b-5a36ee074541","Type":"ContainerStarted","Data":"f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd"}
Nov 02 13:03:45.103261 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.103178    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" event={"ID":"40d9f2c8-011b-49fa-9f7b-5a36ee074541","Type":"ContainerStarted","Data":"3da61e0e02850ef6630e752ffa8bec0ed6edfbfd3d4f92aeb3bdb0c0ff2b8d06"}
Nov 02 13:03:45.169819 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.169752    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" podStartSLOduration=2.169737391 podStartE2EDuration="2.169737391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:03:45.165970801 +0000 UTC m=+2171.272673654" watchObservedRunningTime="2026-11-02 13:03:45.169737391 +0000 UTC m=+2171.276440313"
Nov 02 13:03:45.207157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.207119    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"]
Nov 02 13:03:45.214134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.214098    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-7p92j"]
Nov 02 13:03:45.754235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.754184    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:45.879934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.879892    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:45.886691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:45.886657    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:46.017985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:46.017902    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" path="/var/lib/kubelet/pods/e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5/volumes"
Nov 02 13:03:46.113023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:46.112963    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:46.114143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:46.114114    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:03:47.544271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:47.544207    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:47.662275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:47.662241    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:49.233150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.232941    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:49.367852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.367822    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:49.391625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.391590    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"]
Nov 02 13:03:49.392369 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.392355    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" containerName="istio-proxy"
Nov 02 13:03:49.392434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.392372    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" containerName="istio-proxy"
Nov 02 13:03:49.392546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.392537    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17db8c7-32cf-4a99-9d3c-e87c6b3b69b5" containerName="istio-proxy"
Nov 02 13:03:49.398036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.398009    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"
Nov 02 13:03:49.418010 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.417962    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"]
Nov 02 13:03:49.426676 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.426643    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e3396f\""
Nov 02 13:03:49.426927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.426664    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e3396f-dockercfg-t2nls\""
Nov 02 13:03:49.492219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.492108    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6wc\" (UniqueName: \"kubernetes.io/projected/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-kube-api-access-dw6wc\") pod \"payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc\" (UID: \"5d845a4a-e70b-4b49-b1ea-f3bd4224439b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"
Nov 02 13:03:49.492219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.492173    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc\" (UID: \"5d845a4a-e70b-4b49-b1ea-f3bd4224439b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"
Nov 02 13:03:49.507524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.507484    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"]
Nov 02 13:03:49.526058 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.515624    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"
Nov 02 13:03:49.559019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.551385    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"]
Nov 02 13:03:49.595854 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.595814    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6wc\" (UniqueName: \"kubernetes.io/projected/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-kube-api-access-dw6wc\") pod \"payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc\" (UID: \"5d845a4a-e70b-4b49-b1ea-f3bd4224439b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"
Nov 02 13:03:49.596091 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.595888    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc\" (UID: \"5d845a4a-e70b-4b49-b1ea-f3bd4224439b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"
Nov 02 13:03:49.597039 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.596967    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc\" (UID: \"5d845a4a-e70b-4b49-b1ea-f3bd4224439b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"
Nov 02 13:03:49.621968 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.621931    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6wc\" (UniqueName: \"kubernetes.io/projected/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-kube-api-access-dw6wc\") pod \"payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc\" (UID: \"5d845a4a-e70b-4b49-b1ea-f3bd4224439b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"
Nov 02 13:03:49.697407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.697371    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5c4v\" (UniqueName: \"kubernetes.io/projected/1ca730eb-ca79-4abe-8598-02769a71b915-kube-api-access-q5c4v\") pod \"payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk\" (UID: \"1ca730eb-ca79-4abe-8598-02769a71b915\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"
Nov 02 13:03:49.697407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.697414    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca730eb-ca79-4abe-8598-02769a71b915-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk\" (UID: \"1ca730eb-ca79-4abe-8598-02769a71b915\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"
Nov 02 13:03:49.718325 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.718285    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"
Nov 02 13:03:49.801414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.801065    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5c4v\" (UniqueName: \"kubernetes.io/projected/1ca730eb-ca79-4abe-8598-02769a71b915-kube-api-access-q5c4v\") pod \"payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk\" (UID: \"1ca730eb-ca79-4abe-8598-02769a71b915\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"
Nov 02 13:03:49.801414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.801123    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca730eb-ca79-4abe-8598-02769a71b915-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk\" (UID: \"1ca730eb-ca79-4abe-8598-02769a71b915\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"
Nov 02 13:03:49.802222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.801808    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca730eb-ca79-4abe-8598-02769a71b915-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk\" (UID: \"1ca730eb-ca79-4abe-8598-02769a71b915\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"
Nov 02 13:03:49.827503 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.826852    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5c4v\" (UniqueName: \"kubernetes.io/projected/1ca730eb-ca79-4abe-8598-02769a71b915-kube-api-access-q5c4v\") pod \"payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk\" (UID: \"1ca730eb-ca79-4abe-8598-02769a71b915\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"
Nov 02 13:03:49.876640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:49.876605    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"
Nov 02 13:03:50.037501 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:03:50.037462    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d845a4a_e70b_4b49_b1ea_f3bd4224439b.slice/crio-a2945ad6edafaeb010c2e8b973d00d2ad88126f6d0b53c264bf870b4a6cb6fe5 WatchSource:0}: Error finding container a2945ad6edafaeb010c2e8b973d00d2ad88126f6d0b53c264bf870b4a6cb6fe5: Status 404 returned error can't find the container with id a2945ad6edafaeb010c2e8b973d00d2ad88126f6d0b53c264bf870b4a6cb6fe5
Nov 02 13:03:50.038740 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:50.038693    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"]
Nov 02 13:03:50.151296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:50.151256    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc" event={"ID":"5d845a4a-e70b-4b49-b1ea-f3bd4224439b","Type":"ContainerStarted","Data":"27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270"}
Nov 02 13:03:50.152108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:50.151676    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc" event={"ID":"5d845a4a-e70b-4b49-b1ea-f3bd4224439b","Type":"ContainerStarted","Data":"a2945ad6edafaeb010c2e8b973d00d2ad88126f6d0b53c264bf870b4a6cb6fe5"}
Nov 02 13:03:50.152412 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:50.152380    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"
Nov 02 13:03:50.196180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:50.196126    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc" podStartSLOduration=1.19611292 podStartE2EDuration="1.19611292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:03:50.1938763 +0000 UTC m=+2176.300579160" watchObservedRunningTime="2026-11-02 13:03:50.19611292 +0000 UTC m=+2176.302815772"
Nov 02 13:03:50.210832 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:03:50.210799    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca730eb_ca79_4abe_8598_02769a71b915.slice/crio-d25ec05ecba41a6e0a0e785ef42f648a981b5a9081b8e5850f673edfbc87c2fc WatchSource:0}: Error finding container d25ec05ecba41a6e0a0e785ef42f648a981b5a9081b8e5850f673edfbc87c2fc: Status 404 returned error can't find the container with id d25ec05ecba41a6e0a0e785ef42f648a981b5a9081b8e5850f673edfbc87c2fc
Nov 02 13:03:50.211153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:50.211128    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"]
Nov 02 13:03:50.953871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:50.953821    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:51.160770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:51.160731    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk" event={"ID":"1ca730eb-ca79-4abe-8598-02769a71b915","Type":"ContainerStarted","Data":"2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004"}
Nov 02 13:03:51.161024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:51.160798    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk" event={"ID":"1ca730eb-ca79-4abe-8598-02769a71b915","Type":"ContainerStarted","Data":"d25ec05ecba41a6e0a0e785ef42f648a981b5a9081b8e5850f673edfbc87c2fc"}
Nov 02 13:03:51.161024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:51.160899    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"
Nov 02 13:03:51.204965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:51.204844    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk" podStartSLOduration=2.204829407 podStartE2EDuration="2.204829407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:03:51.203640219 +0000 UTC m=+2177.310343071" watchObservedRunningTime="2026-11-02 13:03:51.204829407 +0000 UTC m=+2177.311532308"
Nov 02 13:03:52.528383 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:52.528338    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:54.232380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:54.232346    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:55.872637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:55.872606    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:57.559874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:57.559840    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:03:59.246420 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:03:59.246388    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:00.970364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:00.970330    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:01.164431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:01.164406    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"
Nov 02 13:04:02.176415 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:02.176385    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"
Nov 02 13:04:02.816501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:02.816464    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:04.651577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:04.651547    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:06.305410 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:06.305359    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:07.996844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:07.996807    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:08.131671 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:08.131635    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:08.261900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:08.261815    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:09.841629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:09.841593    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:11.565321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:11.565282    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:12.581890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:12.581856    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:12.706702 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:12.706666    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:13.255604 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:13.255563    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:14.822474 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:04:14.822435    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:04:14.989170 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:14.989134    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:16.710985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:16.710952    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:18.549519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:18.549486    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:20.289038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:20.288987    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:22.039413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:22.039380    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:23.847382 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:23.847348    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:24.156144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:24.156114    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:24.312048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:24.311974    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:24.446914 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:24.446827    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:24.580857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:24.580828    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:25.212301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:25.212270    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:25.337782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:25.337753    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:25.871186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:25.871156    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:26.135549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:26.135402    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:26.259491 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:26.259447    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:27.638933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.638900    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"]
Nov 02 13:04:27.644495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.644468    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.663799 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.663764    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"]
Nov 02 13:04:27.706645 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.706607    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"agc-cred-38a4a44343e4\""
Nov 02 13:04:27.727398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.727366    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:27.749944 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.749906    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.750149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.749956    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.750208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.750162    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.750261 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.750218    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcttm\" (UniqueName: \"kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-kube-api-access-zcttm\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.851616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.851577    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.851904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.851879    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.852269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.852247    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.852507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.852489    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcttm\" (UniqueName: \"kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-kube-api-access-zcttm\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.853091 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.852712    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.853299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.853266    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.854961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.854943    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.865506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.865480    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:27.877730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.877705    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcttm\" (UniqueName: \"kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-kube-api-access-zcttm\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:27.968790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:27.968700    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:28.278110 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:04:28.278070    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcb4ee3_db9c_496c_9218_25340d29c0fa.slice/crio-1f15420e2681aef066c3e59865ead7fe0c5195f5a6a379aa426cb6950a6f0ece WatchSource:0}: Error finding container 1f15420e2681aef066c3e59865ead7fe0c5195f5a6a379aa426cb6950a6f0ece: Status 404 returned error can't find the container with id 1f15420e2681aef066c3e59865ead7fe0c5195f5a6a379aa426cb6950a6f0ece
Nov 02 13:04:28.278532 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:28.278516    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"]
Nov 02 13:04:28.449694 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:28.449651    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5" event={"ID":"7bcb4ee3-db9c-496c-9218-25340d29c0fa","Type":"ContainerStarted","Data":"d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445"}
Nov 02 13:04:28.449902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:28.449759    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:28.449902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:28.449784    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5" event={"ID":"7bcb4ee3-db9c-496c-9218-25340d29c0fa","Type":"ContainerStarted","Data":"1f15420e2681aef066c3e59865ead7fe0c5195f5a6a379aa426cb6950a6f0ece"}
Nov 02 13:04:28.497254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:28.497187    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5" podStartSLOduration=1.49716962 podStartE2EDuration="1.49716962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:04:28.49505489 +0000 UTC m=+2214.601757741" watchObservedRunningTime="2026-11-02 13:04:28.49716962 +0000 UTC m=+2214.603872473"
Nov 02 13:04:29.474140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:29.474107    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:31.071665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:31.071634    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:32.031111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:32.031081    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"]
Nov 02 13:04:32.044347 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:32.044313    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50f7f72e-xk5lp"]
Nov 02 13:04:32.794693 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:32.794657    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:34.019845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:34.019812    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a0193f-86ca-4a9c-aa04-fd0d80a1743b" path="/var/lib/kubelet/pods/05a0193f-86ca-4a9c-aa04-fd0d80a1743b/volumes"
Nov 02 13:04:34.450774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:34.450734    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:35.049174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:35.049132    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"]
Nov 02 13:04:35.052301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:35.052277    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85ef3ba1-z65j9"]
Nov 02 13:04:36.020505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:36.020470    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572bcf39-cd03-4a12-a5dc-a2e8dca2d872" path="/var/lib/kubelet/pods/572bcf39-cd03-4a12-a5dc-a2e8dca2d872/volumes"
Nov 02 13:04:36.130665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:36.130609    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:37.724102 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:37.724070    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:37.982078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:37.981962    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:39.465606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:39.465567    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:04:39.546499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:39.546461    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"]
Nov 02 13:04:39.770632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:39.770547    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:41.408820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:41.408788    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:43.032385 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:43.032352    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"]
Nov 02 13:04:43.045424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:43.045394    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c831876c-ghbcm"]
Nov 02 13:04:43.095070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:43.095036    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:44.018327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.018294    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cabd776-2d6c-40a2-8abb-661ace6f4de3" path="/var/lib/kubelet/pods/0cabd776-2d6c-40a2-8abb-661ace6f4de3/volumes"
Nov 02 13:04:44.547613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.547573    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c" podUID="0bcd35dc-690c-4034-a5de-f4ce472c9f22" containerName="payload-processing" containerID="cri-o://b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c" gracePeriod=25
Nov 02 13:04:44.824061 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:04:44.824023    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:04:44.827653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.827629    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:04:44.883945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.883909    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-routing-overlay-volume\") pod \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") "
Nov 02 13:04:44.884181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.883973    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:44.884181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.884053    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-plugins-config-volume\") pod \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") "
Nov 02 13:04:44.884181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.884118    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-kube-api-access-sgstd\" (UniqueName: \"kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-kube-api-access-sgstd\") pod \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") "
Nov 02 13:04:44.884389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.884203    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-provider-credentials\") pod \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\" (UID: \"0bcd35dc-690c-4034-a5de-f4ce472c9f22\") "
Nov 02 13:04:44.884389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.884373    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-routing-overlay-volume" pod "0bcd35dc-690c-4034-a5de-f4ce472c9f22" (UID: "0bcd35dc-690c-4034-a5de-f4ce472c9f22"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:04:44.884745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.884724    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-plugins-config-volume" pod "0bcd35dc-690c-4034-a5de-f4ce472c9f22" (UID: "0bcd35dc-690c-4034-a5de-f4ce472c9f22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:04:44.884924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.884907    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:04:44.885035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.884929    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bcd35dc-690c-4034-a5de-f4ce472c9f22-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:04:44.887253 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.887231    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-kube-api-access-sgstd" pod "0bcd35dc-690c-4034-a5de-f4ce472c9f22" (UID: "0bcd35dc-690c-4034-a5de-f4ce472c9f22"). InnerVolumeSpecName "kube-api-access-sgstd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:04:44.887481 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.887458    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-provider-credentials" pod "0bcd35dc-690c-4034-a5de-f4ce472c9f22" (UID: "0bcd35dc-690c-4034-a5de-f4ce472c9f22"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:04:44.986959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.986911    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgstd\" (UniqueName: \"kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-kube-api-access-sgstd\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:04:44.986959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:44.986948    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0bcd35dc-690c-4034-a5de-f4ce472c9f22-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:04:45.031468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:45.031430    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"]
Nov 02 13:04:45.035664 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:45.035633    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2960e17-dcx27"]
Nov 02 13:04:45.587982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:45.587947    5879 generic.go:356] "Generic (PLEG): container finished" podID="0bcd35dc-690c-4034-a5de-f4ce472c9f22" containerID="b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c" exitCode=0
Nov 02 13:04:45.588531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:45.588032    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"
Nov 02 13:04:45.588531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:45.588029    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c" event={"ID":"0bcd35dc-690c-4034-a5de-f4ce472c9f22","Type":"ContainerDied","Data":"b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c"}
Nov 02 13:04:45.588531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:45.588100    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c" event={"ID":"0bcd35dc-690c-4034-a5de-f4ce472c9f22","Type":"ContainerDied","Data":"ca258cf5f384270498586ce7c4f7dbad72125fbe82cdcff65c17a2400a7cca7c"}
Nov 02 13:04:45.588531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:45.588135    5879 scope.go:122] "RemoveContainer" containerID="b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c"
Nov 02 13:04:45.600792 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:45.600767    5879 scope.go:122] "RemoveContainer" containerID="b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c"
Nov 02 13:04:45.601143 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:04:45.601119    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c\": container with ID starting with b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c not found: ID does not exist" containerID="b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c"
Nov 02 13:04:45.601249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:45.601155    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c"} err="failed to get container status \"b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c\": rpc error: code = NotFound desc = could not find container \"b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c\": container with ID starting with b3d024bbd13f846617dc06dbd690860a156a0e35882b88dd56e50f6d6edf170c not found: ID does not exist"
Nov 02 13:04:45.644841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:45.644806    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"]
Nov 02 13:04:45.658363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:45.658331    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f5775xb7c"]
Nov 02 13:04:46.017950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:46.017907    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bcd35dc-690c-4034-a5de-f4ce472c9f22" path="/var/lib/kubelet/pods/0bcd35dc-690c-4034-a5de-f4ce472c9f22/volumes"
Nov 02 13:04:46.018513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:46.018492    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e016209-454b-42ef-9a94-ad72659a1b6a" path="/var/lib/kubelet/pods/9e016209-454b-42ef-9a94-ad72659a1b6a/volumes"
Nov 02 13:04:46.556364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:46.556329    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:48.234882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:48.234853    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:50.002400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:50.002362    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:51.783592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:51.783546    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:52.014287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:52.014254    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:52.812671 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:52.812634    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:53.856695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:53.856659    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:54.016228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:54.016192    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:54.294624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:54.294588    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:54.585477 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:54.585315    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:54.735443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:54.735368    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:54.877205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:54.877167    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:55.424367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:55.424326    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:56.010970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.010935    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:56.436524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.436489    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:56.564247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.564209    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:56.796099 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.796005    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"]
Nov 02 13:04:56.797583 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.797507    5879 cpu_manager.go:414] "containerMap: removing container" podUID="0bcd35dc-690c-4034-a5de-f4ce472c9f22" containerName="payload-processing"
Nov 02 13:04:56.797771 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.797756    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bcd35dc-690c-4034-a5de-f4ce472c9f22" containerName="payload-processing"
Nov 02 13:04:56.798161 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.798143    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bcd35dc-690c-4034-a5de-f4ce472c9f22" containerName="payload-processing"
Nov 02 13:04:56.809497 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.809468    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:04:56.858925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.858838    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"]
Nov 02 13:04:56.898500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.898293    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:04:56.898500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.898467    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:04:56.899915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.899883    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a33c1a-5mmr5\" (UID: \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:04:56.900280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.900255    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a33c1a-5mmr5\" (UID: \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:04:56.900491 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:56.900472    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvjz9\" (UniqueName: \"kubernetes.io/projected/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-kube-api-access-dvjz9\") pod \"maas-api-revoke-sub-40a33c1a-5mmr5\" (UID: \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:04:57.021917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:57.021860    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a33c1a-5mmr5\" (UID: \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:04:57.022486 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:57.022457    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a33c1a-5mmr5\" (UID: \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:04:57.022574 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:57.022549    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvjz9\" (UniqueName: \"kubernetes.io/projected/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-kube-api-access-dvjz9\") pod \"maas-api-revoke-sub-40a33c1a-5mmr5\" (UID: \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:04:57.023578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:57.023553    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a33c1a-5mmr5\" (UID: \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:04:57.026723 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:57.026699    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a33c1a-5mmr5\" (UID: \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:04:57.047541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:57.047474    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvjz9\" (UniqueName: \"kubernetes.io/projected/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-kube-api-access-dvjz9\") pod \"maas-api-revoke-sub-40a33c1a-5mmr5\" (UID: \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:04:57.159468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:57.159433    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:04:57.511587 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:57.511550    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"]
Nov 02 13:04:57.519583 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:04:57.519551    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c07b02b_5bd3_43ac_864b_eb5b5606c05d.slice/crio-9736118eb64a0f923990f17689f1f379fb1138fc187db2e1306cb9fdc4b55548 WatchSource:0}: Error finding container 9736118eb64a0f923990f17689f1f379fb1138fc187db2e1306cb9fdc4b55548: Status 404 returned error can't find the container with id 9736118eb64a0f923990f17689f1f379fb1138fc187db2e1306cb9fdc4b55548
Nov 02 13:04:57.602024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:57.600392    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:57.721020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:57.718458    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5" event={"ID":"5c07b02b-5bd3-43ac-864b-eb5b5606c05d","Type":"ContainerStarted","Data":"9736118eb64a0f923990f17689f1f379fb1138fc187db2e1306cb9fdc4b55548"}
Nov 02 13:04:57.737421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:57.737388    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:57.860720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:57.860642    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:58.063977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.063944    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"]
Nov 02 13:04:58.069713 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.069686    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"]
Nov 02 13:04:58.069962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.069820    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.090073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.087342    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-936529\"/\"agc-cred-2ae2950b3623\""
Nov 02 13:04:58.090073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.087452    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-936529\"/\"openshift-service-ca.crt\""
Nov 02 13:04:58.090073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.087579    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-936529\"/\"kube-root-ca.crt\""
Nov 02 13:04:58.090073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.087645    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-936529\"/\"payload-processing-external-model-plugins-e2e-extmod-b-936529\""
Nov 02 13:04:58.090073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.087736    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-936529\"/\"routing-overlay\""
Nov 02 13:04:58.100218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.100125    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-936529\"/\"payload-processing-external-model-e2e--95eb9016-dockercfg-vtg6n\""
Nov 02 13:04:58.143724 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.143681    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.143885 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.143857    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.144069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.144044    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7x8r\" (UniqueName: \"kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-kube-api-access-n7x8r\") pod \"payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.144123 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.144100    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.239493 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.239453    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:04:58.246194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.246110    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7x8r\" (UniqueName: \"kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-kube-api-access-n7x8r\") pod \"payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.246367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.246222    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.246470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.246410    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.246593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.246575    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.247544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.247512    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.247712 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.247666    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.249474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.249438    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.271776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.271512    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7x8r\" (UniqueName: \"kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-kube-api-access-n7x8r\") pod \"payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.416068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.412196    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:04:58.728949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.728915    5879 generic.go:356] "Generic (PLEG): container finished" podID="5c07b02b-5bd3-43ac-864b-eb5b5606c05d" containerID="32c87fbc1bc9eb50a6c4f1f4e14b47b5364fe4b97a69fa9be56adad4d47de63a" exitCode=0
Nov 02 13:04:58.729333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.729304    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5" event={"ID":"5c07b02b-5bd3-43ac-864b-eb5b5606c05d","Type":"ContainerDied","Data":"32c87fbc1bc9eb50a6c4f1f4e14b47b5364fe4b97a69fa9be56adad4d47de63a"}
Nov 02 13:04:58.730815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.730792    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"]
Nov 02 13:04:58.761898 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:04:58.761872    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2824542d_91bc_4809_b701_3005bf78f2ae.slice/crio-0970a135c52db89d09dd4572cfababf29fb8326f52e047345e1781e2f1527b41 WatchSource:0}: Error finding container 0970a135c52db89d09dd4572cfababf29fb8326f52e047345e1781e2f1527b41: Status 404 returned error can't find the container with id 0970a135c52db89d09dd4572cfababf29fb8326f52e047345e1781e2f1527b41
Nov 02 13:04:58.766229 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.766204    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"]
Nov 02 13:04:58.772076 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.772028    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:58.813072 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.813023    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"agc-cred-26e198286e55\""
Nov 02 13:04:58.818886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.818854    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"]
Nov 02 13:04:58.858769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.858739    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:58.859058 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.859036    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:58.859378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.859357    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcqf9\" (UniqueName: \"kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-kube-api-access-gcqf9\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:58.859558 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.859541    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:58.961481 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.961393    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:58.961625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.961525    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:58.961625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.961584    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcqf9\" (UniqueName: \"kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-kube-api-access-gcqf9\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:58.961734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.961636    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:58.962309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.962277    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:58.962601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.962356    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:58.964300 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:58.964280    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:59.012806 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.012762    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcqf9\" (UniqueName: \"kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-kube-api-access-gcqf9\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:59.142949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.142916    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:59.481340 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:04:59.481299    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1b8103_f660_4f1e_acbd_1bf2e0728ee2.slice/crio-feee1d639a6de378104a7ccaa93af9fd65bdbe4e764a789a19608666ce545ea3 WatchSource:0}: Error finding container feee1d639a6de378104a7ccaa93af9fd65bdbe4e764a789a19608666ce545ea3: Status 404 returned error can't find the container with id feee1d639a6de378104a7ccaa93af9fd65bdbe4e764a789a19608666ce545ea3
Nov 02 13:04:59.481481 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.481457    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"]
Nov 02 13:04:59.624242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.624206    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"]
Nov 02 13:04:59.628855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.628833    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.645297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.645255    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"agc-cred-f63b4fb22333\""
Nov 02 13:04:59.649312 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.649280    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"]
Nov 02 13:04:59.740227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.740129    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55" event={"ID":"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2","Type":"ContainerStarted","Data":"ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f"}
Nov 02 13:04:59.740227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.740191    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55" event={"ID":"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2","Type":"ContainerStarted","Data":"feee1d639a6de378104a7ccaa93af9fd65bdbe4e764a789a19608666ce545ea3"}
Nov 02 13:04:59.740227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.740218    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:04:59.742215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.742185    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc" event={"ID":"2824542d-91bc-4809-b701-3005bf78f2ae","Type":"ContainerStarted","Data":"bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9"}
Nov 02 13:04:59.742339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.742240    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc" event={"ID":"2824542d-91bc-4809-b701-3005bf78f2ae","Type":"ContainerStarted","Data":"0970a135c52db89d09dd4572cfababf29fb8326f52e047345e1781e2f1527b41"}
Nov 02 13:04:59.775030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.774476    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zznml\" (UniqueName: \"kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-kube-api-access-zznml\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.775030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.774596    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.775030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.774705    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.775030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.774816    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.785319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.785251    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55" podStartSLOduration=1.7852336439999998 podStartE2EDuration="1.785233644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:04:59.784226074 +0000 UTC m=+2245.890928926" watchObservedRunningTime="2026-11-02 13:04:59.785233644 +0000 UTC m=+2245.891936498"
Nov 02 13:04:59.848907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.848847    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc" podStartSLOduration=2.848826185 podStartE2EDuration="2.848826185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:04:59.829144493 +0000 UTC m=+2245.935847345" watchObservedRunningTime="2026-11-02 13:04:59.848826185 +0000 UTC m=+2245.955529037"
Nov 02 13:04:59.876035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.875954    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.879758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.876514    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zznml\" (UniqueName: \"kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-kube-api-access-zznml\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.879758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.876601    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.879758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.876743    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.879758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.877503    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.879758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.878663    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.883834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.883807    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.901092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.901065    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:04:59.904697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.904668    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zznml\" (UniqueName: \"kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-kube-api-access-zznml\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.951550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.951505    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:04:59.984351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.984313    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-openshift-ca\") pod \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\" (UID: \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\") "
Nov 02 13:04:59.984589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.984569    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-xks-ca\") pod \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\" (UID: \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\") "
Nov 02 13:04:59.984656 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.984634    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-kube-api-access-dvjz9\" (UniqueName: \"kubernetes.io/projected/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-kube-api-access-dvjz9\") pod \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\" (UID: \"5c07b02b-5bd3-43ac-864b-eb5b5606c05d\") "
Nov 02 13:04:59.985109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.985085    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-openshift-ca" pod "5c07b02b-5bd3-43ac-864b-eb5b5606c05d" (UID: "5c07b02b-5bd3-43ac-864b-eb5b5606c05d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:04:59.985360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.985341    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:04:59.987621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.987598    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-xks-ca" pod "5c07b02b-5bd3-43ac-864b-eb5b5606c05d" (UID: "5c07b02b-5bd3-43ac-864b-eb5b5606c05d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:04:59.987621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:04:59.987605    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-kube-api-access-dvjz9" pod "5c07b02b-5bd3-43ac-864b-eb5b5606c05d" (UID: "5c07b02b-5bd3-43ac-864b-eb5b5606c05d"). InnerVolumeSpecName "kube-api-access-dvjz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:05:00.087814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:00.087776    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:00.087814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:00.087817    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvjz9\" (UniqueName: \"kubernetes.io/projected/5c07b02b-5bd3-43ac-864b-eb5b5606c05d-kube-api-access-dvjz9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:00.109795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:00.109761    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:00.277295 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:05:00.277243    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1b6305_0e83_4366_be0d_98931c0d224f.slice/crio-70d2edfed977969f7c1c50f0703e45c1b1aae0d74b1c419ed502dc1f9b3856cf WatchSource:0}: Error finding container 70d2edfed977969f7c1c50f0703e45c1b1aae0d74b1c419ed502dc1f9b3856cf: Status 404 returned error can't find the container with id 70d2edfed977969f7c1c50f0703e45c1b1aae0d74b1c419ed502dc1f9b3856cf
Nov 02 13:05:00.278147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:00.278116    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"]
Nov 02 13:05:00.753555 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:00.753509    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"
Nov 02 13:05:00.753555 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:00.753536    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5" event={"ID":"5c07b02b-5bd3-43ac-864b-eb5b5606c05d","Type":"ContainerDied","Data":"9736118eb64a0f923990f17689f1f379fb1138fc187db2e1306cb9fdc4b55548"}
Nov 02 13:05:00.753858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:00.753592    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9736118eb64a0f923990f17689f1f379fb1138fc187db2e1306cb9fdc4b55548"
Nov 02 13:05:00.755754 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:00.755723    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf" event={"ID":"bd1b6305-0e83-4366-be0d-98931c0d224f","Type":"ContainerStarted","Data":"5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e"}
Nov 02 13:05:00.756257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:00.755780    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf" event={"ID":"bd1b6305-0e83-4366-be0d-98931c0d224f","Type":"ContainerStarted","Data":"70d2edfed977969f7c1c50f0703e45c1b1aae0d74b1c419ed502dc1f9b3856cf"}
Nov 02 13:05:00.756527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:00.756505    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:05:00.756660 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:00.756554    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:05:00.800195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:00.800131    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf" podStartSLOduration=1.800115333 podStartE2EDuration="1.800115333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:05:00.799128824 +0000 UTC m=+2246.905831675" watchObservedRunningTime="2026-11-02 13:05:00.800115333 +0000 UTC m=+2246.906818182"
Nov 02 13:05:01.963925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:01.963884    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:02.046840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:02.043096    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"]
Nov 02 13:05:02.055312 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:02.055273    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"]
Nov 02 13:05:02.059419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:02.059387    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42de8aea-95mx5"]
Nov 02 13:05:02.063137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:02.063115    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d5860d8-wkv7q"]
Nov 02 13:05:02.154214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:02.154179    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:05:02.154388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:02.154243    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:05:03.736380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:03.736236    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:03.785584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:03.785552    5879 generic.go:356] "Generic (PLEG): container finished" podID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerID="f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd" exitCode=0
Nov 02 13:05:03.785771 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:03.785625    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" event={"ID":"40d9f2c8-011b-49fa-9f7b-5a36ee074541","Type":"ContainerDied","Data":"f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd"}
Nov 02 13:05:03.786150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:03.786132    5879 scope.go:122] "RemoveContainer" containerID="f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd"
Nov 02 13:05:03.788197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:03.788164    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:05:03.788322 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:03.788248    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:05:03.788322 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:03.788312    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:05:04.019714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:04.019630    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c030f6fc-22c2-4f8d-aa39-4337a17ebd4d" path="/var/lib/kubelet/pods/c030f6fc-22c2-4f8d-aa39-4337a17ebd4d/volumes"
Nov 02 13:05:04.020174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:04.020160    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90a6ac9-e592-410a-ae4c-aa4f95e9f596" path="/var/lib/kubelet/pods/e90a6ac9-e592-410a-ae4c-aa4f95e9f596/volumes"
Nov 02 13:05:04.814345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:04.814295    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" event={"ID":"40d9f2c8-011b-49fa-9f7b-5a36ee074541","Type":"ContainerStarted","Data":"a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f"}
Nov 02 13:05:04.879970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:04.879929    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:05:04.880219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:04.880199    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:05:04.880268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:04.880245    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Nov 02 13:05:05.457209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:05.457172    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:05.880526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:05.880493    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:05:05.881083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:05.880552    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Nov 02 13:05:06.880451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:06.880418    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:05:06.880713 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:06.880471    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Nov 02 13:05:07.263438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:07.263340    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:07.880121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:07.880085    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:05:07.880327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:07.880144    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Nov 02 13:05:08.102182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:08.102145    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"]
Nov 02 13:05:08.132618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:08.132521    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:08.249053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:08.249005    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:08.750752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:08.750714    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:08.880313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:08.880277    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:05:08.880529 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:08.880348    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Nov 02 13:05:09.296802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:09.296598    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:09.880064 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:09.880029    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body=
Nov 02 13:05:09.880255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:09.880080    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused"
Nov 02 13:05:10.759379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:10.759340    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:05:10.839941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:10.839908    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"]
Nov 02 13:05:10.885820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:10.885771    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:05:10.886915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:10.886885    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:05:10.887842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:10.887816    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:05:11.276808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:11.276779    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:11.767657 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:11.767626    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:05:12.776878 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:12.776844    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:05:12.856609 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:12.856569    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"]
Nov 02 13:05:13.032106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.032014    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"]
Nov 02 13:05:13.036353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.036321    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"]
Nov 02 13:05:13.037489 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.037463    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:13.039776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.039755    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e378974b-45dwr"]
Nov 02 13:05:13.043092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.043074    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a4812e-5xfd8"]
Nov 02 13:05:13.104075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.103947    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8" podUID="b0cc0d39-8a32-4852-9267-c4b3926b5685" containerName="payload-processing" containerID="cri-o://11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855" gracePeriod=25
Nov 02 13:05:13.399564 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.399538    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:05:13.408809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.408782    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-plugins-config-volume\") pod \"b0cc0d39-8a32-4852-9267-c4b3926b5685\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") "
Nov 02 13:05:13.408982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.408902    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-kube-api-access-hpm97\" (UniqueName: \"kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-kube-api-access-hpm97\") pod \"b0cc0d39-8a32-4852-9267-c4b3926b5685\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") "
Nov 02 13:05:13.409209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.409187    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-plugins-config-volume" pod "b0cc0d39-8a32-4852-9267-c4b3926b5685" (UID: "b0cc0d39-8a32-4852-9267-c4b3926b5685"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:05:13.409507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.409488    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:13.411651 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.411617    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-kube-api-access-hpm97" pod "b0cc0d39-8a32-4852-9267-c4b3926b5685" (UID: "b0cc0d39-8a32-4852-9267-c4b3926b5685"). InnerVolumeSpecName "kube-api-access-hpm97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:05:13.510872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.510796    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-routing-overlay-volume\") pod \"b0cc0d39-8a32-4852-9267-c4b3926b5685\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") "
Nov 02 13:05:13.511076 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.510955    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-provider-credentials\") pod \"b0cc0d39-8a32-4852-9267-c4b3926b5685\" (UID: \"b0cc0d39-8a32-4852-9267-c4b3926b5685\") "
Nov 02 13:05:13.511258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.511230    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-routing-overlay-volume" pod "b0cc0d39-8a32-4852-9267-c4b3926b5685" (UID: "b0cc0d39-8a32-4852-9267-c4b3926b5685"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:05:13.511661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.511642    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b0cc0d39-8a32-4852-9267-c4b3926b5685-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:13.511703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.511666    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpm97\" (UniqueName: \"kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-kube-api-access-hpm97\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:13.513357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.513336    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-provider-credentials" pod "b0cc0d39-8a32-4852-9267-c4b3926b5685" (UID: "b0cc0d39-8a32-4852-9267-c4b3926b5685"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:05:13.613155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.613068    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b0cc0d39-8a32-4852-9267-c4b3926b5685-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:13.901439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.901402    5879 generic.go:356] "Generic (PLEG): container finished" podID="b0cc0d39-8a32-4852-9267-c4b3926b5685" containerID="11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855" exitCode=0
Nov 02 13:05:13.902038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.901450    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"
Nov 02 13:05:13.902038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.901465    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8" event={"ID":"b0cc0d39-8a32-4852-9267-c4b3926b5685","Type":"ContainerDied","Data":"11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855"}
Nov 02 13:05:13.902038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.901521    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8" event={"ID":"b0cc0d39-8a32-4852-9267-c4b3926b5685","Type":"ContainerDied","Data":"11d80203f067ebe22e77d018f06213af85406ccfe4203f79ba93380dffd4de0a"}
Nov 02 13:05:13.902038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.901552    5879 scope.go:122] "RemoveContainer" containerID="11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855"
Nov 02 13:05:13.912978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.912953    5879 scope.go:122] "RemoveContainer" containerID="11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855"
Nov 02 13:05:13.913328 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:05:13.913299    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855\": container with ID starting with 11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855 not found: ID does not exist" containerID="11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855"
Nov 02 13:05:13.913401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.913330    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855"} err="failed to get container status \"11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855\": rpc error: code = NotFound desc = could not find container \"11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855\": container with ID starting with 11a1add9d1c7a346e8bf3006f4fe4bd35f9acc1cab783779a502b3d59bef9855 not found: ID does not exist"
Nov 02 13:05:13.960755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.960713    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"]
Nov 02 13:05:13.974735 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:13.974694    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0e422b/payload-processing-external-model-e2e-extmod-b-0e422b-6bcbp55k8"]
Nov 02 13:05:14.021635 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:14.021602    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1082fd85-ff1f-4183-ba9f-a0549c57adc2" path="/var/lib/kubelet/pods/1082fd85-ff1f-4183-ba9f-a0549c57adc2/volumes"
Nov 02 13:05:14.022273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:14.022250    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0cc0d39-8a32-4852-9267-c4b3926b5685" path="/var/lib/kubelet/pods/b0cc0d39-8a32-4852-9267-c4b3926b5685/volumes"
Nov 02 13:05:14.022743 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:14.022723    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f256cf5a-c275-4b6a-b0ae-8af247ad5355" path="/var/lib/kubelet/pods/f256cf5a-c275-4b6a-b0ae-8af247ad5355/volumes"
Nov 02 13:05:14.514409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:14.514383    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:14.823412 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:05:14.823311    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:05:14.968279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:14.968247    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:15.096727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:15.096639    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:15.238594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:15.238553    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:15.666196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:15.666160    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:15.787984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:15.787952    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:15.841362 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:15.841325    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp" podUID="2a5b9b5f-2237-46aa-b406-88f551af2789" containerName="payload-processing" containerID="cri-o://22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29" gracePeriod=25
Nov 02 13:05:15.959917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:15.959894    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:16.124620 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.124589    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:05:16.250305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.250220    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-plugins-config-volume\") pod \"2a5b9b5f-2237-46aa-b406-88f551af2789\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") "
Nov 02 13:05:16.250476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.250323    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-routing-overlay-volume\") pod \"2a5b9b5f-2237-46aa-b406-88f551af2789\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") "
Nov 02 13:05:16.250476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.250432    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-provider-credentials\") pod \"2a5b9b5f-2237-46aa-b406-88f551af2789\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") "
Nov 02 13:05:16.251193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.250546    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-kube-api-access-w4v9g\" (UniqueName: \"kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-kube-api-access-w4v9g\") pod \"2a5b9b5f-2237-46aa-b406-88f551af2789\" (UID: \"2a5b9b5f-2237-46aa-b406-88f551af2789\") "
Nov 02 13:05:16.251193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.250673    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-plugins-config-volume" pod "2a5b9b5f-2237-46aa-b406-88f551af2789" (UID: "2a5b9b5f-2237-46aa-b406-88f551af2789"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:05:16.251193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.250668    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-routing-overlay-volume" pod "2a5b9b5f-2237-46aa-b406-88f551af2789" (UID: "2a5b9b5f-2237-46aa-b406-88f551af2789"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:05:16.251193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.251147    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:16.251193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.251170    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a5b9b5f-2237-46aa-b406-88f551af2789-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:16.252978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.252956    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-kube-api-access-w4v9g" pod "2a5b9b5f-2237-46aa-b406-88f551af2789" (UID: "2a5b9b5f-2237-46aa-b406-88f551af2789"). InnerVolumeSpecName "kube-api-access-w4v9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:05:16.253108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.252987    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-provider-credentials" pod "2a5b9b5f-2237-46aa-b406-88f551af2789" (UID: "2a5b9b5f-2237-46aa-b406-88f551af2789"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:05:16.277341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.277310    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:16.352751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.352719    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:16.352751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.352750    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4v9g\" (UniqueName: \"kubernetes.io/projected/2a5b9b5f-2237-46aa-b406-88f551af2789-kube-api-access-w4v9g\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:16.403611 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.403573    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:16.529591 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.529485    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:16.659782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.659752    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:16.930854 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.930817    5879 generic.go:356] "Generic (PLEG): container finished" podID="2a5b9b5f-2237-46aa-b406-88f551af2789" containerID="22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29" exitCode=0
Nov 02 13:05:16.931097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.930869    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"
Nov 02 13:05:16.931097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.930904    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp" event={"ID":"2a5b9b5f-2237-46aa-b406-88f551af2789","Type":"ContainerDied","Data":"22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29"}
Nov 02 13:05:16.931097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.930958    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp" event={"ID":"2a5b9b5f-2237-46aa-b406-88f551af2789","Type":"ContainerDied","Data":"f1c28759c8c5728291ee78adcd839e0a4300b28719f80d32302719ac6fddfd3b"}
Nov 02 13:05:16.931097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.931024    5879 scope.go:122] "RemoveContainer" containerID="22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29"
Nov 02 13:05:16.943842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.943820    5879 scope.go:122] "RemoveContainer" containerID="22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29"
Nov 02 13:05:16.944190 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:05:16.944172    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29\": container with ID starting with 22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29 not found: ID does not exist" containerID="22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29"
Nov 02 13:05:16.944296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.944198    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29"} err="failed to get container status \"22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29\": rpc error: code = NotFound desc = could not find container \"22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29\": container with ID starting with 22be480ea16b9bfd3e3d1f43e0bed1c05f5e37c9ed6921f3ab54a51791a27d29 not found: ID does not exist"
Nov 02 13:05:16.990294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:16.990251    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"]
Nov 02 13:05:17.002599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:17.002567    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578f977cp"]
Nov 02 13:05:17.079464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:17.079434    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:17.225966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:17.225887    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:17.857043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:17.856985    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5" podUID="7bcb4ee3-db9c-496c-9218-25340d29c0fa" containerName="payload-processing" containerID="cri-o://d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445" gracePeriod=25
Nov 02 13:05:18.020134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.020093    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a5b9b5f-2237-46aa-b406-88f551af2789" path="/var/lib/kubelet/pods/2a5b9b5f-2237-46aa-b406-88f551af2789/volumes"
Nov 02 13:05:18.141290 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.141266    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:05:18.179260 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.179228    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-provider-credentials\") pod \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") "
Nov 02 13:05:18.179260 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.179265    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-routing-overlay-volume\") pod \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") "
Nov 02 13:05:18.179549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.179455    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-plugins-config-volume\") pod \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") "
Nov 02 13:05:18.179611 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.179556    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-kube-api-access-zcttm\" (UniqueName: \"kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-kube-api-access-zcttm\") pod \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\" (UID: \"7bcb4ee3-db9c-496c-9218-25340d29c0fa\") "
Nov 02 13:05:18.179797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.179681    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-routing-overlay-volume" pod "7bcb4ee3-db9c-496c-9218-25340d29c0fa" (UID: "7bcb4ee3-db9c-496c-9218-25340d29c0fa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:05:18.179900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.179881    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-plugins-config-volume" pod "7bcb4ee3-db9c-496c-9218-25340d29c0fa" (UID: "7bcb4ee3-db9c-496c-9218-25340d29c0fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:05:18.180498 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.180476    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:18.180578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.180505    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bcb4ee3-db9c-496c-9218-25340d29c0fa-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:18.181886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.181865    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-kube-api-access-zcttm" pod "7bcb4ee3-db9c-496c-9218-25340d29c0fa" (UID: "7bcb4ee3-db9c-496c-9218-25340d29c0fa"). InnerVolumeSpecName "kube-api-access-zcttm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:05:18.182050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.182028    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-provider-credentials" pod "7bcb4ee3-db9c-496c-9218-25340d29c0fa" (UID: "7bcb4ee3-db9c-496c-9218-25340d29c0fa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:05:18.282222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.282191    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcttm\" (UniqueName: \"kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-kube-api-access-zcttm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:18.282222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.282216    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bcb4ee3-db9c-496c-9218-25340d29c0fa-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:18.525351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.525318    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:18.949778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.949741    5879 generic.go:356] "Generic (PLEG): container finished" podID="7bcb4ee3-db9c-496c-9218-25340d29c0fa" containerID="d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445" exitCode=0
Nov 02 13:05:18.949963 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.949787    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"
Nov 02 13:05:18.949963 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.949820    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5" event={"ID":"7bcb4ee3-db9c-496c-9218-25340d29c0fa","Type":"ContainerDied","Data":"d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445"}
Nov 02 13:05:18.949963 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.949873    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5" event={"ID":"7bcb4ee3-db9c-496c-9218-25340d29c0fa","Type":"ContainerDied","Data":"1f15420e2681aef066c3e59865ead7fe0c5195f5a6a379aa426cb6950a6f0ece"}
Nov 02 13:05:18.949963 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.949898    5879 scope.go:122] "RemoveContainer" containerID="d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445"
Nov 02 13:05:18.961152 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.961135    5879 scope.go:122] "RemoveContainer" containerID="d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445"
Nov 02 13:05:18.961438 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:05:18.961419    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445\": container with ID starting with d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445 not found: ID does not exist" containerID="d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445"
Nov 02 13:05:18.961518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:18.961453    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445"} err="failed to get container status \"d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445\": rpc error: code = NotFound desc = could not find container \"d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445\": container with ID starting with d6e0b8b9916b64f8cd66c3b0e060bff2084686765f3deaf555e5c90957d69445 not found: ID does not exist"
Nov 02 13:05:19.006809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:19.006764    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"]
Nov 02 13:05:19.020058 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:19.020021    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-77965gsc5"]
Nov 02 13:05:20.018542 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:20.018506    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bcb4ee3-db9c-496c-9218-25340d29c0fa" path="/var/lib/kubelet/pods/7bcb4ee3-db9c-496c-9218-25340d29c0fa/volumes"
Nov 02 13:05:20.297176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:20.297087    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:21.445598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:21.445565    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:21.556054 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:21.556021    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:21.913827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:21.913795    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:22.085285 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:22.085243    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:22.231918 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:22.231831    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:23.007314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.007279    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:23.251471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.251439    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"]
Nov 02 13:05:23.252327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.252303    5879 cpu_manager.go:414] "containerMap: removing container" podUID="7bcb4ee3-db9c-496c-9218-25340d29c0fa" containerName="payload-processing"
Nov 02 13:05:23.252327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.252326    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bcb4ee3-db9c-496c-9218-25340d29c0fa" containerName="payload-processing"
Nov 02 13:05:23.252549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.252364    5879 cpu_manager.go:414] "containerMap: removing container" podUID="b0cc0d39-8a32-4852-9267-c4b3926b5685" containerName="payload-processing"
Nov 02 13:05:23.252549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.252371    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0cc0d39-8a32-4852-9267-c4b3926b5685" containerName="payload-processing"
Nov 02 13:05:23.252549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.252394    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5c07b02b-5bd3-43ac-864b-eb5b5606c05d" containerName="revoke-keys"
Nov 02 13:05:23.252549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.252402    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c07b02b-5bd3-43ac-864b-eb5b5606c05d" containerName="revoke-keys"
Nov 02 13:05:23.252549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.252436    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2a5b9b5f-2237-46aa-b406-88f551af2789" containerName="payload-processing"
Nov 02 13:05:23.252549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.252443    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5b9b5f-2237-46aa-b406-88f551af2789" containerName="payload-processing"
Nov 02 13:05:23.252853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.252602    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c07b02b-5bd3-43ac-864b-eb5b5606c05d" containerName="revoke-keys"
Nov 02 13:05:23.252853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.252618    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5b9b5f-2237-46aa-b406-88f551af2789" containerName="payload-processing"
Nov 02 13:05:23.252853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.252636    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bcb4ee3-db9c-496c-9218-25340d29c0fa" containerName="payload-processing"
Nov 02 13:05:23.252853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.252650    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0cc0d39-8a32-4852-9267-c4b3926b5685" containerName="payload-processing"
Nov 02 13:05:23.256539 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.256512    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.276786 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.276749    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"]
Nov 02 13:05:23.286163 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.286130    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25f7a6\"/\"e2e-cmp-legacy-5df383-provider-cred\""
Nov 02 13:05:23.286341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.286173    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25f7a6\"/\"payload-processing-external-model-e2e--edace3a4-dockercfg-r8sd2\""
Nov 02 13:05:23.286341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.286179    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25f7a6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-25f7a6\""
Nov 02 13:05:23.286341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.286227    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25f7a6\"/\"routing-overlay\""
Nov 02 13:05:23.288339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.288287    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25f7a6\"/\"kube-root-ca.crt\""
Nov 02 13:05:23.288339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.288300    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-25f7a6\"/\"openshift-service-ca.crt\""
Nov 02 13:05:23.353552 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.353514    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.353552 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.353556    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.353793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.353733    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5g5\" (UniqueName: \"kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-kube-api-access-lh5g5\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.353862 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.353847    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.455365 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.455326    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5g5\" (UniqueName: \"kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-kube-api-access-lh5g5\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.455605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.455476    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.455753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.455735    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.455832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.455772    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.456568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.456537    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.456715 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.456624    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.458566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.458536    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.480987 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.480952    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5g5\" (UniqueName: \"kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-kube-api-access-lh5g5\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.570843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.570737    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:23.881335 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:05:23.881286    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1fcba3_2a1c_45bd_8845_eba530ce578b.slice/crio-02f77e6c2ebaf458a77f81385489bce74266a7790792bd43def4f4a0c9c79be8 WatchSource:0}: Error finding container 02f77e6c2ebaf458a77f81385489bce74266a7790792bd43def4f4a0c9c79be8: Status 404 returned error can't find the container with id 02f77e6c2ebaf458a77f81385489bce74266a7790792bd43def4f4a0c9c79be8
Nov 02 13:05:23.881538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.881514    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"]
Nov 02 13:05:23.888737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.888712    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:23.991067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.991029    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr" event={"ID":"1f1fcba3-2a1c-45bd-8845-eba530ce578b","Type":"ContainerStarted","Data":"c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950"}
Nov 02 13:05:23.991242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.991093    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr" event={"ID":"1f1fcba3-2a1c-45bd-8845-eba530ce578b","Type":"ContainerStarted","Data":"02f77e6c2ebaf458a77f81385489bce74266a7790792bd43def4f4a0c9c79be8"}
Nov 02 13:05:23.991242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:23.991119    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:24.037071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:24.037024    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr" podStartSLOduration=1.037010714 podStartE2EDuration="1.037010714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:05:24.03683515 +0000 UTC m=+2270.143538001" watchObservedRunningTime="2026-11-02 13:05:24.037010714 +0000 UTC m=+2270.143713567"
Nov 02 13:05:25.657487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:25.657438    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:27.442490 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:27.442454    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:29.177487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:29.177446    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:29.479750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:29.479660    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:29.607166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:29.607135    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:30.052317 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:30.052282    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:30.945797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:30.945767    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:32.627409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:32.627378    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:34.396827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:34.396795    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:35.003466 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:35.003434    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:05:36.048775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:36.048745    5879 scope.go:122] "RemoveContainer" containerID="ab1f7d63a0dd4eddcd9ef13b55c72829fde53c9d2090429619c968705c58eeb4"
Nov 02 13:05:36.056634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:36.056604    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:36.059593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:36.059573    5879 scope.go:122] "RemoveContainer" containerID="e1325d3484a90445e72f6ad09220cf16d25729e9f3e56a3e7c853a2d08db1a37"
Nov 02 13:05:36.342227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:36.342114    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:36.482953 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:36.482924    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:36.659065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:36.659022    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:37.859078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:37.859044    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:37.981013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:37.980973    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:39.679803 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:39.679769    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:41.516695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:41.516625    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:42.970965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:42.970929    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:43.103441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:43.103410    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:43.245394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:43.245278    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:43.467387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:43.467346    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:43.590345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:43.590228    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:43.708702 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:43.708640    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:43.907203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:43.907157    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:44.697410 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:44.697380    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:44.829279 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:05:44.829240    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:05:44.935377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:44.935341    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"]
Nov 02 13:05:44.941941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:44.941919    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:44.959851 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:44.959827    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"]
Nov 02 13:05:44.969766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:44.969742    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"openshift-service-ca.crt\""
Nov 02 13:05:44.970024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:44.969977    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"payload-processing-external-model-plugins-e2e-extmod-a-740f09\""
Nov 02 13:05:44.970161 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:44.970146    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"kube-root-ca.crt\""
Nov 02 13:05:44.970620 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:44.970600    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"routing-overlay\""
Nov 02 13:05:44.970723 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:44.970650    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"agc-cred-3af3e654ec18\""
Nov 02 13:05:44.975235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:44.975213    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"payload-processing-external-model-e2e--49869bd7-dockercfg-dx8tk\""
Nov 02 13:05:45.066616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.066575    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7897s\" (UniqueName: \"kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-kube-api-access-7897s\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.066794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.066622    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.066794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.066685    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.066886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.066831    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.168857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.168815    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.169104 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.168951    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.169459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.169435    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7897s\" (UniqueName: \"kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-kube-api-access-7897s\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.169649 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.169628    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.169867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.169839    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.170656 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.170632    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.172334 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.172310    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.195076 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.195041    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7897s\" (UniqueName: \"kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-kube-api-access-7897s\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.256708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.256646    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:45.407870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.407840    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:45.560060 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:05:45.560028    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e64c37_43b7_4893_b680_e0af16f58442.slice/crio-35e91d00d3a107ce466515544476cac30b179ce2a9810a181885661ce8415dc8 WatchSource:0}: Error finding container 35e91d00d3a107ce466515544476cac30b179ce2a9810a181885661ce8415dc8: Status 404 returned error can't find the container with id 35e91d00d3a107ce466515544476cac30b179ce2a9810a181885661ce8415dc8
Nov 02 13:05:45.560256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.560236    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"]
Nov 02 13:05:45.736968 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.736935    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"]
Nov 02 13:05:45.741445 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.741426    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:45.759065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.759036    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"agc-cred-3e2202435795\""
Nov 02 13:05:45.763259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.763231    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"]
Nov 02 13:05:45.880276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.880247    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnhtw\" (UniqueName: \"kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-kube-api-access-qnhtw\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:45.880446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.880383    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:45.880505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.880494    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:45.880550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.880516    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:45.977871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.977839    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:45.981579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.981548    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:45.981767 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.981714    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:45.981767 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.981740    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:45.981914 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.981810    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnhtw\" (UniqueName: \"kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-kube-api-access-qnhtw\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:45.982441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.982414    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:45.982548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.982497    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:45.984850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:45.984821    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:46.007218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:46.007177    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnhtw\" (UniqueName: \"kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-kube-api-access-qnhtw\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:46.056094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:46.056058    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:46.204394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:46.204307    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8" event={"ID":"79e64c37-43b7-4893-b680-e0af16f58442","Type":"ContainerStarted","Data":"f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07"}
Nov 02 13:05:46.204394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:46.204356    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8" event={"ID":"79e64c37-43b7-4893-b680-e0af16f58442","Type":"ContainerStarted","Data":"35e91d00d3a107ce466515544476cac30b179ce2a9810a181885661ce8415dc8"}
Nov 02 13:05:46.204641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:46.204447    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:46.252655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:46.252597    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8" podStartSLOduration=2.252581898 podStartE2EDuration="2.252581898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:05:46.249724594 +0000 UTC m=+2292.356427445" watchObservedRunningTime="2026-11-02 13:05:46.252581898 +0000 UTC m=+2292.359284750"
Nov 02 13:05:46.367254 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:05:46.367207    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f9c526_0a62_43aa_a9c1_14efa35acbd8.slice/crio-81c7660980cf1030a9ca8a2b6a384a97d2e200e38d01e97e63e0015f1612aa3d WatchSource:0}: Error finding container 81c7660980cf1030a9ca8a2b6a384a97d2e200e38d01e97e63e0015f1612aa3d: Status 404 returned error can't find the container with id 81c7660980cf1030a9ca8a2b6a384a97d2e200e38d01e97e63e0015f1612aa3d
Nov 02 13:05:46.369003 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:46.368967    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"]
Nov 02 13:05:47.089484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.089446    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:47.217597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.217534    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8" event={"ID":"e0f9c526-0a62-43aa-a9c1-14efa35acbd8","Type":"ContainerStarted","Data":"564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41"}
Nov 02 13:05:47.217964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.217943    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8" event={"ID":"e0f9c526-0a62-43aa-a9c1-14efa35acbd8","Type":"ContainerStarted","Data":"81c7660980cf1030a9ca8a2b6a384a97d2e200e38d01e97e63e0015f1612aa3d"}
Nov 02 13:05:47.218222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.218207    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:47.267042 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.266976    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8" podStartSLOduration=2.266961802 podStartE2EDuration="2.266961802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:05:47.263480097 +0000 UTC m=+2293.370182948" watchObservedRunningTime="2026-11-02 13:05:47.266961802 +0000 UTC m=+2293.373664653"
Nov 02 13:05:47.343869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.343780    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:47.723128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.723080    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"]
Nov 02 13:05:47.728492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.728472    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:47.749420 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.749382    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"]
Nov 02 13:05:47.807947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.807915    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:47.839523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.839483    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:47.839697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.839533    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4h7t\" (UniqueName: \"kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-kube-api-access-b4h7t\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:47.839697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.839666    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:47.839786 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.839767    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:47.937020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.934405    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:47.948016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.941772    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:47.948016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.943366    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:47.948016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.943429    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4h7t\" (UniqueName: \"kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-kube-api-access-b4h7t\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:47.948016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.943509    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:47.948016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.944172    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:47.948016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.944802    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:47.948016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.947809    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:47.974870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:47.971202    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4h7t\" (UniqueName: \"kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-kube-api-access-b4h7t\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:48.049041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:48.049006    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:48.358598 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:05:48.358571    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a9550d_09b4_47eb_b614_069f0f66186b.slice/crio-a48b8e8a88e814702ac6d85368d08763253699bcd3fd1752c717647deeb3bbd0 WatchSource:0}: Error finding container a48b8e8a88e814702ac6d85368d08763253699bcd3fd1752c717647deeb3bbd0: Status 404 returned error can't find the container with id a48b8e8a88e814702ac6d85368d08763253699bcd3fd1752c717647deeb3bbd0
Nov 02 13:05:48.359665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:48.359631    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"]
Nov 02 13:05:49.120687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.120657    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:49.233041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.232975    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj" event={"ID":"31a9550d-09b4-47eb-b614-069f0f66186b","Type":"ContainerStarted","Data":"eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266"}
Nov 02 13:05:49.233266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.233068    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj" event={"ID":"31a9550d-09b4-47eb-b614-069f0f66186b","Type":"ContainerStarted","Data":"a48b8e8a88e814702ac6d85368d08763253699bcd3fd1752c717647deeb3bbd0"}
Nov 02 13:05:49.233266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.233104    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:05:49.281849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.281797    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj" podStartSLOduration=2.281784066 podStartE2EDuration="2.281784066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:05:49.280413799 +0000 UTC m=+2295.387116650" watchObservedRunningTime="2026-11-02 13:05:49.281784066 +0000 UTC m=+2295.388486917"
Nov 02 13:05:49.589214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.589130    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:49.833812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.833780    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"]
Nov 02 13:05:49.838370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.838349    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:49.857109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.857083    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"openshift-service-ca.crt\""
Nov 02 13:05:49.857313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.857094    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-14fcb6\""
Nov 02 13:05:49.859153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.859128    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"]
Nov 02 13:05:49.868208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.868184    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"e2e-cfl-4ead53-provider-cred\""
Nov 02 13:05:49.868353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.868190    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"routing-overlay\""
Nov 02 13:05:49.868353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.868198    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"kube-root-ca.crt\""
Nov 02 13:05:49.868353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.868230    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"payload-processing-external-model-e2e--ee70ab17-dockercfg-xg2rj\""
Nov 02 13:05:49.884659 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.884627    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:49.884835 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.884795    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:49.885001 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.884964    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvt7c\" (UniqueName: \"kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-kube-api-access-bvt7c\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:49.885254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.885235    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:49.987108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.987064    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvt7c\" (UniqueName: \"kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-kube-api-access-bvt7c\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:49.987301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.987226    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:49.987375 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.987330    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:49.987446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.987397    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:49.988198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.988160    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:49.988358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.988266    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:49.990077 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:49.990053    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:50.044146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:50.044099    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvt7c\" (UniqueName: \"kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-kube-api-access-bvt7c\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:50.154900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:50.154863    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:50.482486 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:50.482456    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"]
Nov 02 13:05:50.482891 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:05:50.482864    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d729291_dafd_48a9_aa6d_1d8f06f0d729.slice/crio-a709604e38692596f45e25ec91ad98bfd05ab9ac62868d92fa76119f469358f4 WatchSource:0}: Error finding container a709604e38692596f45e25ec91ad98bfd05ab9ac62868d92fa76119f469358f4: Status 404 returned error can't find the container with id a709604e38692596f45e25ec91ad98bfd05ab9ac62868d92fa76119f469358f4
Nov 02 13:05:50.980213 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:50.980173    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:51.190945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:51.190912    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"]
Nov 02 13:05:51.193589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:51.193565    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:51.252716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:51.252649    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw" event={"ID":"5d729291-dafd-48a9-aa6d-1d8f06f0d729","Type":"ContainerStarted","Data":"b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af"}
Nov 02 13:05:51.252716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:51.252694    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw" event={"ID":"5d729291-dafd-48a9-aa6d-1d8f06f0d729","Type":"ContainerStarted","Data":"a709604e38692596f45e25ec91ad98bfd05ab9ac62868d92fa76119f469358f4"}
Nov 02 13:05:51.252891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:51.252734    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:05:51.280578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:51.280532    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:51.326068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:51.325983    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw" podStartSLOduration=2.3259707880000002 podStartE2EDuration="2.325970788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:05:51.322340557 +0000 UTC m=+2297.429043410" watchObservedRunningTime="2026-11-02 13:05:51.325970788 +0000 UTC m=+2297.432673662"
Nov 02 13:05:51.781019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:51.780958    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:52.763191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:52.763153    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:54.433137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:54.433079    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:56.191345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.191307    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8" podUID="79e64c37-43b7-4893-b680-e0af16f58442" containerName="payload-processing" containerID="cri-o://f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07" gracePeriod=25
Nov 02 13:05:56.274842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.274815    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:56.478126 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.478100    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:56.571936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.571907    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-provider-credentials\" (UniqueName: \"kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-provider-credentials\") pod \"79e64c37-43b7-4893-b680-e0af16f58442\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") "
Nov 02 13:05:56.572128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.571977    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-routing-overlay-volume\") pod \"79e64c37-43b7-4893-b680-e0af16f58442\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") "
Nov 02 13:05:56.572128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.572063    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-kube-api-access-7897s\" (UniqueName: \"kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-kube-api-access-7897s\") pod \"79e64c37-43b7-4893-b680-e0af16f58442\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") "
Nov 02 13:05:56.572256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.572228    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-plugins-config-volume\") pod \"79e64c37-43b7-4893-b680-e0af16f58442\" (UID: \"79e64c37-43b7-4893-b680-e0af16f58442\") "
Nov 02 13:05:56.572396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.572373    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-routing-overlay-volume" pod "79e64c37-43b7-4893-b680-e0af16f58442" (UID: "79e64c37-43b7-4893-b680-e0af16f58442"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:05:56.572661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.572619    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-plugins-config-volume" pod "79e64c37-43b7-4893-b680-e0af16f58442" (UID: "79e64c37-43b7-4893-b680-e0af16f58442"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:05:56.572905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.572889    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:56.572982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.572907    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79e64c37-43b7-4893-b680-e0af16f58442-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:56.574916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.574890    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-provider-credentials" pod "79e64c37-43b7-4893-b680-e0af16f58442" (UID: "79e64c37-43b7-4893-b680-e0af16f58442"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:05:56.574916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.574901    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-kube-api-access-7897s" pod "79e64c37-43b7-4893-b680-e0af16f58442" (UID: "79e64c37-43b7-4893-b680-e0af16f58442"). InnerVolumeSpecName "kube-api-access-7897s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:05:56.674474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.674445    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7897s\" (UniqueName: \"kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-kube-api-access-7897s\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:56.674474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:56.674472    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79e64c37-43b7-4893-b680-e0af16f58442-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:05:57.093634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:57.093603    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:57.278638 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:57.278604    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:57.311053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:57.310987    5879 generic.go:356] "Generic (PLEG): container finished" podID="79e64c37-43b7-4893-b680-e0af16f58442" containerID="f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07" exitCode=0
Nov 02 13:05:57.311319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:57.311289    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8" event={"ID":"79e64c37-43b7-4893-b680-e0af16f58442","Type":"ContainerDied","Data":"f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07"}
Nov 02 13:05:57.311519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:57.311495    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8" event={"ID":"79e64c37-43b7-4893-b680-e0af16f58442","Type":"ContainerDied","Data":"35e91d00d3a107ce466515544476cac30b179ce2a9810a181885661ce8415dc8"}
Nov 02 13:05:57.311650 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:57.311536    5879 scope.go:122] "RemoveContainer" containerID="f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07"
Nov 02 13:05:57.312437 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:57.312420    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"
Nov 02 13:05:57.343151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:57.343122    5879 scope.go:122] "RemoveContainer" containerID="f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07"
Nov 02 13:05:57.343924 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:05:57.343659    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07\": container with ID starting with f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07 not found: ID does not exist" containerID="f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07"
Nov 02 13:05:57.343924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:57.343699    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07"} err="failed to get container status \"f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07\": rpc error: code = NotFound desc = could not find container \"f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07\": container with ID starting with f3702db12e9a1ddec286408255873015f61965f0fc2d5355c72ac14910340a07 not found: ID does not exist"
Nov 02 13:05:57.385487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:57.385461    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"]
Nov 02 13:05:57.398836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:57.398811    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-8496p7ph8"]
Nov 02 13:05:58.018731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:58.018698    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e64c37-43b7-4893-b680-e0af16f58442" path="/var/lib/kubelet/pods/79e64c37-43b7-4893-b680-e0af16f58442/volumes"
Nov 02 13:05:58.156499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:58.156456    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:58.279886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:58.279808    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:05:59.236870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:59.236839    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:05:59.316357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:59.316329    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"]
Nov 02 13:05:59.858046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:05:59.858011    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:00.247194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:00.247167    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:06:00.324386 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:00.324356    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"]
Nov 02 13:06:01.584839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:01.584799    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:02.263433 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:02.263403    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:06:03.375665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:03.375632    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:04.316956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.316913    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55" podUID="2a1b8103-f660-4f1e-acbd-1bf2e0728ee2" containerName="payload-processing" containerID="cri-o://ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f" gracePeriod=25
Nov 02 13:06:04.593631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.593607    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:06:04.701333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.701302    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-plugins-config-volume\") pod \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") "
Nov 02 13:06:04.701487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.701441    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-kube-api-access-gcqf9\" (UniqueName: \"kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-kube-api-access-gcqf9\") pod \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") "
Nov 02 13:06:04.701559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.701535    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-provider-credentials\") pod \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") "
Nov 02 13:06:04.701619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.701602    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-routing-overlay-volume\") pod \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\" (UID: \"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2\") "
Nov 02 13:06:04.701796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.701772    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-plugins-config-volume" pod "2a1b8103-f660-4f1e-acbd-1bf2e0728ee2" (UID: "2a1b8103-f660-4f1e-acbd-1bf2e0728ee2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:04.702013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.701969    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-routing-overlay-volume" pod "2a1b8103-f660-4f1e-acbd-1bf2e0728ee2" (UID: "2a1b8103-f660-4f1e-acbd-1bf2e0728ee2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:04.702282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.702174    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:04.702282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.702191    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:04.703716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.703697    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-kube-api-access-gcqf9" pod "2a1b8103-f660-4f1e-acbd-1bf2e0728ee2" (UID: "2a1b8103-f660-4f1e-acbd-1bf2e0728ee2"). InnerVolumeSpecName "kube-api-access-gcqf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:04.703835 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.703822    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-provider-credentials" pod "2a1b8103-f660-4f1e-acbd-1bf2e0728ee2" (UID: "2a1b8103-f660-4f1e-acbd-1bf2e0728ee2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:04.803448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.803412    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcqf9\" (UniqueName: \"kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-kube-api-access-gcqf9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:04.803600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.803453    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:04.807975 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.807939    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:04.921264 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:04.921231    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:05.250196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.250100    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:05.325379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.325326    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf" podUID="bd1b6305-0e83-4366-be0d-98931c0d224f" containerName="payload-processing" containerID="cri-o://5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e" gracePeriod=25
Nov 02 13:06:05.382960 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.382919    5879 generic.go:356] "Generic (PLEG): container finished" podID="2a1b8103-f660-4f1e-acbd-1bf2e0728ee2" containerID="ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f" exitCode=0
Nov 02 13:06:05.383159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.382985    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"
Nov 02 13:06:05.383159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.383070    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55" event={"ID":"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2","Type":"ContainerDied","Data":"ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f"}
Nov 02 13:06:05.383159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.383119    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55" event={"ID":"2a1b8103-f660-4f1e-acbd-1bf2e0728ee2","Type":"ContainerDied","Data":"feee1d639a6de378104a7ccaa93af9fd65bdbe4e764a789a19608666ce545ea3"}
Nov 02 13:06:05.383159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.383147    5879 scope.go:122] "RemoveContainer" containerID="ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f"
Nov 02 13:06:05.393233 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.393191    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:05.443690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.443665    5879 scope.go:122] "RemoveContainer" containerID="ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f"
Nov 02 13:06:05.444327 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:06:05.444202    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f\": container with ID starting with ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f not found: ID does not exist" containerID="ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f"
Nov 02 13:06:05.444327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.444248    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f"} err="failed to get container status \"ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f\": rpc error: code = NotFound desc = could not find container \"ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f\": container with ID starting with ecdb9bfa6ae5a6ad9c29f47e1475be26c11b92c588fcff2160e46955919ed90f not found: ID does not exist"
Nov 02 13:06:05.444982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.444961    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"]
Nov 02 13:06:05.458591 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.458567    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfdmz55"]
Nov 02 13:06:05.542880 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.542786    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:05.657897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.657859    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:06:05.720925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.718838    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-kube-api-access-zznml\" (UniqueName: \"kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-kube-api-access-zznml\") pod \"bd1b6305-0e83-4366-be0d-98931c0d224f\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") "
Nov 02 13:06:05.720925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.718913    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-plugins-config-volume\") pod \"bd1b6305-0e83-4366-be0d-98931c0d224f\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") "
Nov 02 13:06:05.720925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.718947    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-provider-credentials\") pod \"bd1b6305-0e83-4366-be0d-98931c0d224f\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") "
Nov 02 13:06:05.720925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.719061    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-routing-overlay-volume\") pod \"bd1b6305-0e83-4366-be0d-98931c0d224f\" (UID: \"bd1b6305-0e83-4366-be0d-98931c0d224f\") "
Nov 02 13:06:05.720925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.720058    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-plugins-config-volume" pod "bd1b6305-0e83-4366-be0d-98931c0d224f" (UID: "bd1b6305-0e83-4366-be0d-98931c0d224f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:05.720925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.720207    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-routing-overlay-volume" pod "bd1b6305-0e83-4366-be0d-98931c0d224f" (UID: "bd1b6305-0e83-4366-be0d-98931c0d224f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:05.722380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.722351    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-provider-credentials" pod "bd1b6305-0e83-4366-be0d-98931c0d224f" (UID: "bd1b6305-0e83-4366-be0d-98931c0d224f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:05.722584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.722565    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-kube-api-access-zznml" pod "bd1b6305-0e83-4366-be0d-98931c0d224f" (UID: "bd1b6305-0e83-4366-be0d-98931c0d224f"). InnerVolumeSpecName "kube-api-access-zznml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:05.821057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.820960    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:05.821057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.821017    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:05.821057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.821035    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zznml\" (UniqueName: \"kubernetes.io/projected/bd1b6305-0e83-4366-be0d-98931c0d224f-kube-api-access-zznml\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:05.821057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:05.821050    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd1b6305-0e83-4366-be0d-98931c0d224f-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:06.019211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.019178    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a1b8103-f660-4f1e-acbd-1bf2e0728ee2" path="/var/lib/kubelet/pods/2a1b8103-f660-4f1e-acbd-1bf2e0728ee2/volumes"
Nov 02 13:06:06.391244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.391213    5879 generic.go:356] "Generic (PLEG): container finished" podID="bd1b6305-0e83-4366-be0d-98931c0d224f" containerID="5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e" exitCode=0
Nov 02 13:06:06.391444 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.391270    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"
Nov 02 13:06:06.391444 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.391299    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf" event={"ID":"bd1b6305-0e83-4366-be0d-98931c0d224f","Type":"ContainerDied","Data":"5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e"}
Nov 02 13:06:06.391444 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.391393    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf" event={"ID":"bd1b6305-0e83-4366-be0d-98931c0d224f","Type":"ContainerDied","Data":"70d2edfed977969f7c1c50f0703e45c1b1aae0d74b1c419ed502dc1f9b3856cf"}
Nov 02 13:06:06.391444 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.391421    5879 scope.go:122] "RemoveContainer" containerID="5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e"
Nov 02 13:06:06.401756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.401738    5879 scope.go:122] "RemoveContainer" containerID="5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e"
Nov 02 13:06:06.402034 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:06:06.401977    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e\": container with ID starting with 5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e not found: ID does not exist" containerID="5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e"
Nov 02 13:06:06.402158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.402040    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e"} err="failed to get container status \"5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e\": rpc error: code = NotFound desc = could not find container \"5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e\": container with ID starting with 5f916ecf9d93e7845b4cacacc1b242d2f64930fd4c6a3731a9b7103e4c1a1e9e not found: ID does not exist"
Nov 02 13:06:06.450480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.450456    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"]
Nov 02 13:06:06.464440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.464419    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7cf5kf5bf"]
Nov 02 13:06:06.467438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.467420    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"]
Nov 02 13:06:06.468405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.468383    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2a1b8103-f660-4f1e-acbd-1bf2e0728ee2" containerName="payload-processing"
Nov 02 13:06:06.468405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.468403    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a1b8103-f660-4f1e-acbd-1bf2e0728ee2" containerName="payload-processing"
Nov 02 13:06:06.468541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.468452    5879 cpu_manager.go:414] "containerMap: removing container" podUID="79e64c37-43b7-4893-b680-e0af16f58442" containerName="payload-processing"
Nov 02 13:06:06.468541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.468459    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e64c37-43b7-4893-b680-e0af16f58442" containerName="payload-processing"
Nov 02 13:06:06.468541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.468476    5879 cpu_manager.go:414] "containerMap: removing container" podUID="bd1b6305-0e83-4366-be0d-98931c0d224f" containerName="payload-processing"
Nov 02 13:06:06.468541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.468481    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd1b6305-0e83-4366-be0d-98931c0d224f" containerName="payload-processing"
Nov 02 13:06:06.468665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.468576    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd1b6305-0e83-4366-be0d-98931c0d224f" containerName="payload-processing"
Nov 02 13:06:06.468665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.468584    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a1b8103-f660-4f1e-acbd-1bf2e0728ee2" containerName="payload-processing"
Nov 02 13:06:06.468665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.468598    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e64c37-43b7-4893-b680-e0af16f58442" containerName="payload-processing"
Nov 02 13:06:06.473834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.473816    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.488052 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.488032    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"]
Nov 02 13:06:06.490488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.490462    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-936529\"/\"agc-cred-fb150566aed2\""
Nov 02 13:06:06.529939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.529906    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.530118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.530102    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.530172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.530135    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78qw9\" (UniqueName: \"kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-kube-api-access-78qw9\") pod \"payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.530335 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.530313    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.632114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.632084    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.632292 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.632123    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.632292 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.632237    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.632292 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.632263    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78qw9\" (UniqueName: \"kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-kube-api-access-78qw9\") pod \"payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.632893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.632825    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.632893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.632897    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.634740 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.634717    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.657932 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.657877    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78qw9\" (UniqueName: \"kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-kube-api-access-78qw9\") pod \"payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") " pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:06.789764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:06.789733    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:07.098063 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:06:07.098028    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f60268_a57b_426f_8900_f77464006263.slice/crio-64266d3315543fbf2c19180068df5aa227e1699b65ea5fa6a963639869e0a1b1 WatchSource:0}: Error finding container 64266d3315543fbf2c19180068df5aa227e1699b65ea5fa6a963639869e0a1b1: Status 404 returned error can't find the container with id 64266d3315543fbf2c19180068df5aa227e1699b65ea5fa6a963639869e0a1b1
Nov 02 13:06:07.098529 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.098507    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"]
Nov 02 13:06:07.270462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.270164    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:07.400164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.400133    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:07.402851 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.402823    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq" event={"ID":"91f60268-a57b-426f-8900-f77464006263","Type":"ContainerStarted","Data":"7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77"}
Nov 02 13:06:07.403031 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.402877    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq" event={"ID":"91f60268-a57b-426f-8900-f77464006263","Type":"ContainerStarted","Data":"64266d3315543fbf2c19180068df5aa227e1699b65ea5fa6a963639869e0a1b1"}
Nov 02 13:06:07.403103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.403046    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:07.446935 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.446890    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq" podStartSLOduration=1.446876916 podStartE2EDuration="1.446876916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:06:07.445614548 +0000 UTC m=+2313.552317399" watchObservedRunningTime="2026-11-02 13:06:07.446876916 +0000 UTC m=+2313.553579804"
Nov 02 13:06:07.646213 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.646185    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"]
Nov 02 13:06:07.651186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.651130    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.670277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.670256    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ef5efe\""
Nov 02 13:06:07.670277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.670259    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"routing-overlay\""
Nov 02 13:06:07.671650 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.671615    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"agc-cred-d34b16923f21\""
Nov 02 13:06:07.672076 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.672038    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"openshift-service-ca.crt\""
Nov 02 13:06:07.672180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.672131    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"kube-root-ca.crt\""
Nov 02 13:06:07.672180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.672131    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"payload-processing-external-model-e2e--00beae95-dockercfg-jkqxm\""
Nov 02 13:06:07.673330 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.673309    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"]
Nov 02 13:06:07.748303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.748275    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.748417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.748318    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.748417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.748350    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjtn\" (UniqueName: \"kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-kube-api-access-rmjtn\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.748510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.748434    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.856972 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.851178    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.856972 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.851737    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.856972 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.851792    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjtn\" (UniqueName: \"kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-kube-api-access-rmjtn\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.856972 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.851821    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.856972 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.852837    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.856972 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.855503    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.860321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.860294    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.879688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.879661    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjtn\" (UniqueName: \"kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-kube-api-access-rmjtn\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:07.966286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:07.966211    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:08.021722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:08.021690    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd1b6305-0e83-4366-be0d-98931c0d224f" path="/var/lib/kubelet/pods/bd1b6305-0e83-4366-be0d-98931c0d224f/volumes"
Nov 02 13:06:08.272793 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:06:08.272762    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9e49d2_7281_43cf_b8ec_ea065bf1a5dc.slice/crio-c607b6d6bfde396f9e93078ef4dae0f9773be149ca5bc31953f295253089c2d1 WatchSource:0}: Error finding container c607b6d6bfde396f9e93078ef4dae0f9773be149ca5bc31953f295253089c2d1: Status 404 returned error can't find the container with id c607b6d6bfde396f9e93078ef4dae0f9773be149ca5bc31953f295253089c2d1
Nov 02 13:06:08.273729 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:08.273706    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"]
Nov 02 13:06:08.412520 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:08.412484    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t" event={"ID":"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc","Type":"ContainerStarted","Data":"d43fa8b6b4b61a1b1b6f9fbd0bf65fee14cd864da0d12f0ef7ab3bb412ad772f"}
Nov 02 13:06:08.412679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:08.412540    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t" event={"ID":"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc","Type":"ContainerStarted","Data":"c607b6d6bfde396f9e93078ef4dae0f9773be149ca5bc31953f295253089c2d1"}
Nov 02 13:06:08.412679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:08.412580    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:08.456753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:08.456710    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t" podStartSLOduration=1.4566974529999999 podStartE2EDuration="1.456697453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:06:08.455337955 +0000 UTC m=+2314.562040806" watchObservedRunningTime="2026-11-02 13:06:08.456697453 +0000 UTC m=+2314.563400303"
Nov 02 13:06:09.049348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:09.049291    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:10.883496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:10.883461    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:11.243946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:11.243851    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:11.668219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:11.668144    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:11.886723 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:11.886689    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"]
Nov 02 13:06:11.892616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:11.892592    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:11.910394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:11.910363    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"]
Nov 02 13:06:12.002795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.002720    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.002956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.002937    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsz4w\" (UniqueName: \"kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-kube-api-access-zsz4w\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.003049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.002974    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.003188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.003160    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.104502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.104464    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.104686 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.104579    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.105120 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.104830    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsz4w\" (UniqueName: \"kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-kube-api-access-zsz4w\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.105120 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.104878    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.105637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.105611    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.105926 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.105905    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.107666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.107637    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.130812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.130783    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsz4w\" (UniqueName: \"kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-kube-api-access-zsz4w\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.183384 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.183355    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:12.212527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.212497    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:12.309634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.309482    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:12.533238 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:06:12.533174    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48bffff_84c9_4c96_a141_4bb96945d21d.slice/crio-e7b928e588c8f45e2a20b8917e4cb1353956f89250a1a0ec01431ea09ed4e639 WatchSource:0}: Error finding container e7b928e588c8f45e2a20b8917e4cb1353956f89250a1a0ec01431ea09ed4e639: Status 404 returned error can't find the container with id e7b928e588c8f45e2a20b8917e4cb1353956f89250a1a0ec01431ea09ed4e639
Nov 02 13:06:12.534938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.534915    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"]
Nov 02 13:06:12.747630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.747593    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:12.758439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.758397    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"]
Nov 02 13:06:12.761531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.761510    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:12.936166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:12.936138    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:13.285085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:13.284979    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"]
Nov 02 13:06:13.288529 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:13.288504    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:13.384714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:13.384673    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:13.455431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:13.455396    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd" event={"ID":"e48bffff-84c9-4c96-a141-4bb96945d21d","Type":"ContainerStarted","Data":"d63a426b52678e4684db7c1b1b48a54fc3514a8572572c3eda17a0cda7ff6378"}
Nov 02 13:06:13.455655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:13.455463    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:13.455655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:13.455486    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd" event={"ID":"e48bffff-84c9-4c96-a141-4bb96945d21d","Type":"ContainerStarted","Data":"e7b928e588c8f45e2a20b8917e4cb1353956f89250a1a0ec01431ea09ed4e639"}
Nov 02 13:06:13.503804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:13.503747    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd" podStartSLOduration=2.503731418 podStartE2EDuration="2.503731418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:06:13.499197335 +0000 UTC m=+2319.605900216" watchObservedRunningTime="2026-11-02 13:06:13.503731418 +0000 UTC m=+2319.610434271"
Nov 02 13:06:14.693747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:14.693712    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:14.824433 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:06:14.824398    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:06:14.825306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:14.825287    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:16.479790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:16.479737    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:17.759462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:17.759394    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq" podUID="91f60268-a57b-426f-8900-f77464006263" containerName="payload-processing" containerID="cri-o://7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77" gracePeriod=25
Nov 02 13:06:18.036499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.036469    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:18.189696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.189666    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-routing-overlay-volume\") pod \"91f60268-a57b-426f-8900-f77464006263\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") "
Nov 02 13:06:18.189906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.189733    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-provider-credentials\" (UniqueName: \"kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-provider-credentials\") pod \"91f60268-a57b-426f-8900-f77464006263\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") "
Nov 02 13:06:18.189906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.189817    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-kube-api-access-78qw9\" (UniqueName: \"kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-kube-api-access-78qw9\") pod \"91f60268-a57b-426f-8900-f77464006263\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") "
Nov 02 13:06:18.190137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.189919    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-plugins-config-volume\") pod \"91f60268-a57b-426f-8900-f77464006263\" (UID: \"91f60268-a57b-426f-8900-f77464006263\") "
Nov 02 13:06:18.190137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.190111    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-routing-overlay-volume" pod "91f60268-a57b-426f-8900-f77464006263" (UID: "91f60268-a57b-426f-8900-f77464006263"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:18.190460 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.190434    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-plugins-config-volume" pod "91f60268-a57b-426f-8900-f77464006263" (UID: "91f60268-a57b-426f-8900-f77464006263"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:18.190725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.190643    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:18.190725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.190663    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91f60268-a57b-426f-8900-f77464006263-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:18.192343 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.192321    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-provider-credentials" pod "91f60268-a57b-426f-8900-f77464006263" (UID: "91f60268-a57b-426f-8900-f77464006263"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:18.192450 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.192377    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-kube-api-access-78qw9" pod "91f60268-a57b-426f-8900-f77464006263" (UID: "91f60268-a57b-426f-8900-f77464006263"). InnerVolumeSpecName "kube-api-access-78qw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:18.286328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.286231    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t" podUID="4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc" containerName="payload-processing" containerID="cri-o://d43fa8b6b4b61a1b1b6f9fbd0bf65fee14cd864da0d12f0ef7ab3bb412ad772f" gracePeriod=25
Nov 02 13:06:18.292494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.292464    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:18.292494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.292494    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78qw9\" (UniqueName: \"kubernetes.io/projected/91f60268-a57b-426f-8900-f77464006263-kube-api-access-78qw9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:18.495597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.495561    5879 generic.go:356] "Generic (PLEG): container finished" podID="91f60268-a57b-426f-8900-f77464006263" containerID="7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77" exitCode=0
Nov 02 13:06:18.495739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.495661    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"
Nov 02 13:06:18.495739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.495693    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq" event={"ID":"91f60268-a57b-426f-8900-f77464006263","Type":"ContainerDied","Data":"7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77"}
Nov 02 13:06:18.495850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.495749    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq" event={"ID":"91f60268-a57b-426f-8900-f77464006263","Type":"ContainerDied","Data":"64266d3315543fbf2c19180068df5aa227e1699b65ea5fa6a963639869e0a1b1"}
Nov 02 13:06:18.495850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.495778    5879 scope.go:122] "RemoveContainer" containerID="7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77"
Nov 02 13:06:18.498072 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.498044    5879 generic.go:356] "Generic (PLEG): container finished" podID="4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc" containerID="d43fa8b6b4b61a1b1b6f9fbd0bf65fee14cd864da0d12f0ef7ab3bb412ad772f" exitCode=0
Nov 02 13:06:18.498212 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.498072    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t" event={"ID":"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc","Type":"ContainerDied","Data":"d43fa8b6b4b61a1b1b6f9fbd0bf65fee14cd864da0d12f0ef7ab3bb412ad772f"}
Nov 02 13:06:18.509757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.509625    5879 scope.go:122] "RemoveContainer" containerID="7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77"
Nov 02 13:06:18.509956 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:06:18.509934    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77\": container with ID starting with 7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77 not found: ID does not exist" containerID="7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77"
Nov 02 13:06:18.510079 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.509966    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77"} err="failed to get container status \"7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77\": rpc error: code = NotFound desc = could not find container \"7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77\": container with ID starting with 7f9ea5c6af7c19ce90ec3fd50b62e4ee3144e1606f15819bfab9c37f005eea77 not found: ID does not exist"
Nov 02 13:06:18.519739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.519717    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:18.542441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.542354    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:18.558234 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.558195    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"]
Nov 02 13:06:18.570487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.570461    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-7db72ktxq"]
Nov 02 13:06:18.697233 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.697204    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-provider-credentials\") pod \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") "
Nov 02 13:06:18.697429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.697259    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-plugins-config-volume\") pod \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") "
Nov 02 13:06:18.697511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.697494    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-kube-api-access-rmjtn\" (UniqueName: \"kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-kube-api-access-rmjtn\") pod \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") "
Nov 02 13:06:18.697569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.697545    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-routing-overlay-volume\") pod \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\" (UID: \"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc\") "
Nov 02 13:06:18.697754 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.697728    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-plugins-config-volume" pod "4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc" (UID: "4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:18.697973 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.697950    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-routing-overlay-volume" pod "4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc" (UID: "4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:18.698145 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.698128    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:18.698189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.698148    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:18.699658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.699632    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-provider-credentials" pod "4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc" (UID: "4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:18.699745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.699710    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-kube-api-access-rmjtn" pod "4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc" (UID: "4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc"). InnerVolumeSpecName "kube-api-access-rmjtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:18.799667 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.799612    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmjtn\" (UniqueName: \"kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-kube-api-access-rmjtn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:18.799667 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:18.799638    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:19.506889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.506858    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"
Nov 02 13:06:19.507126 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.506857    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t" event={"ID":"4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc","Type":"ContainerDied","Data":"c607b6d6bfde396f9e93078ef4dae0f9773be149ca5bc31953f295253089c2d1"}
Nov 02 13:06:19.507126 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.507038    5879 scope.go:122] "RemoveContainer" containerID="d43fa8b6b4b61a1b1b6f9fbd0bf65fee14cd864da0d12f0ef7ab3bb412ad772f"
Nov 02 13:06:19.570597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.570561    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"]
Nov 02 13:06:19.577474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.577452    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59d8gqv2t"]
Nov 02 13:06:19.952543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.952507    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"]
Nov 02 13:06:19.953353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.953327    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc" containerName="payload-processing"
Nov 02 13:06:19.953353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.953352    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc" containerName="payload-processing"
Nov 02 13:06:19.953502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.953377    5879 cpu_manager.go:414] "containerMap: removing container" podUID="91f60268-a57b-426f-8900-f77464006263" containerName="payload-processing"
Nov 02 13:06:19.953502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.953383    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f60268-a57b-426f-8900-f77464006263" containerName="payload-processing"
Nov 02 13:06:19.953502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.953491    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc" containerName="payload-processing"
Nov 02 13:06:19.953502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.953503    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f60268-a57b-426f-8900-f77464006263" containerName="payload-processing"
Nov 02 13:06:19.958594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.958573    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:19.976335 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.976310    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:06:19.978687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.978667    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:06:19.978868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:19.978847    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"]
Nov 02 13:06:20.020366 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.020337    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc" path="/var/lib/kubelet/pods/4b9e49d2-7281-43cf-b8ec-ea065bf1a5dc/volumes"
Nov 02 13:06:20.020738 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.020725    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f60268-a57b-426f-8900-f77464006263" path="/var/lib/kubelet/pods/91f60268-a57b-426f-8900-f77464006263/volumes"
Nov 02 13:06:20.114548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.114518    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bd0e71-jtqc5\" (UID: \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:20.114705 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.114675    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bd0e71-jtqc5\" (UID: \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:20.114769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.114704    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48mcq\" (UniqueName: \"kubernetes.io/projected/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-kube-api-access-48mcq\") pod \"maas-api-revoke-sub-d2bd0e71-jtqc5\" (UID: \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:20.216115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.216028    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bd0e71-jtqc5\" (UID: \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:20.216282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.216173    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bd0e71-jtqc5\" (UID: \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:20.216282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.216210    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48mcq\" (UniqueName: \"kubernetes.io/projected/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-kube-api-access-48mcq\") pod \"maas-api-revoke-sub-d2bd0e71-jtqc5\" (UID: \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:20.216858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.216837    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bd0e71-jtqc5\" (UID: \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:20.218506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.218488    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bd0e71-jtqc5\" (UID: \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:20.247833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.247812    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:20.253672 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.253639    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48mcq\" (UniqueName: \"kubernetes.io/projected/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-kube-api-access-48mcq\") pod \"maas-api-revoke-sub-d2bd0e71-jtqc5\" (UID: \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:20.273359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.273330    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:20.450396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.450365    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:20.582356 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:06:20.582314    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb9fa63_8ef6_4d2b_829b_be78b96a5a4c.slice/crio-f58bf25263c0686b37993d15ce30bab83943f457fa488fa271b8e4f7d3cdf0ca WatchSource:0}: Error finding container f58bf25263c0686b37993d15ce30bab83943f457fa488fa271b8e4f7d3cdf0ca: Status 404 returned error can't find the container with id f58bf25263c0686b37993d15ce30bab83943f457fa488fa271b8e4f7d3cdf0ca
Nov 02 13:06:20.583732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.583706    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"]
Nov 02 13:06:20.770836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.770798    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:20.899488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:20.899461    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:21.526858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:21.526820    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5" event={"ID":"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c","Type":"ContainerStarted","Data":"8449eaef0c15b4068ec369c8ca816ab11cb2d15cc7907fa55f22c6e719db6bf3"}
Nov 02 13:06:21.527262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:21.526881    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5" event={"ID":"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c","Type":"ContainerStarted","Data":"f58bf25263c0686b37993d15ce30bab83943f457fa488fa271b8e4f7d3cdf0ca"}
Nov 02 13:06:21.583948 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:21.583899    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5" podStartSLOduration=2.583886267 podStartE2EDuration="2.583886267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:06:21.583381499 +0000 UTC m=+2327.690084350" watchObservedRunningTime="2026-11-02 13:06:21.583886267 +0000 UTC m=+2327.690589117"
Nov 02 13:06:22.278926 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.278892    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"]
Nov 02 13:06:22.283362 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.283346    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:22.292548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.292526    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:22.302225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.302206    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"]
Nov 02 13:06:22.450742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.450713    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2716652-2mlg2\" (UID: \"594df756-ef67-40a2-b480-4168ec6c7ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:22.450956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.450776    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2716652-2mlg2\" (UID: \"594df756-ef67-40a2-b480-4168ec6c7ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:22.450956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.450923    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnjz2\" (UniqueName: \"kubernetes.io/projected/594df756-ef67-40a2-b480-4168ec6c7ac0-kube-api-access-cnjz2\") pod \"maas-api-revoke-sub-b2716652-2mlg2\" (UID: \"594df756-ef67-40a2-b480-4168ec6c7ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:22.536319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.536239    5879 generic.go:356] "Generic (PLEG): container finished" podID="7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c" containerID="8449eaef0c15b4068ec369c8ca816ab11cb2d15cc7907fa55f22c6e719db6bf3" exitCode=0
Nov 02 13:06:22.536319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.536289    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5" event={"ID":"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c","Type":"ContainerDied","Data":"8449eaef0c15b4068ec369c8ca816ab11cb2d15cc7907fa55f22c6e719db6bf3"}
Nov 02 13:06:22.552671 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.552641    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2716652-2mlg2\" (UID: \"594df756-ef67-40a2-b480-4168ec6c7ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:22.552797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.552696    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnjz2\" (UniqueName: \"kubernetes.io/projected/594df756-ef67-40a2-b480-4168ec6c7ac0-kube-api-access-cnjz2\") pod \"maas-api-revoke-sub-b2716652-2mlg2\" (UID: \"594df756-ef67-40a2-b480-4168ec6c7ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:22.552865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.552851    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2716652-2mlg2\" (UID: \"594df756-ef67-40a2-b480-4168ec6c7ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:22.553309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.553289    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2716652-2mlg2\" (UID: \"594df756-ef67-40a2-b480-4168ec6c7ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:22.555345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.555321    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2716652-2mlg2\" (UID: \"594df756-ef67-40a2-b480-4168ec6c7ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:22.585436 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.585412    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnjz2\" (UniqueName: \"kubernetes.io/projected/594df756-ef67-40a2-b480-4168ec6c7ac0-kube-api-access-cnjz2\") pod \"maas-api-revoke-sub-b2716652-2mlg2\" (UID: \"594df756-ef67-40a2-b480-4168ec6c7ac0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:22.598438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.598419    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:22.910708 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:06:22.910675    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594df756_ef67_40a2_b480_4168ec6c7ac0.slice/crio-45da959819f02e827b732f7e6ac372dd9e6ae77a67889907effc978e87852410 WatchSource:0}: Error finding container 45da959819f02e827b732f7e6ac372dd9e6ae77a67889907effc978e87852410: Status 404 returned error can't find the container with id 45da959819f02e827b732f7e6ac372dd9e6ae77a67889907effc978e87852410
Nov 02 13:06:22.912294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:22.912273    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"]
Nov 02 13:06:23.545146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.545052    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2" event={"ID":"594df756-ef67-40a2-b480-4168ec6c7ac0","Type":"ContainerStarted","Data":"0e22335c6ad5d2714e07b4b8b4ce754593d556177ad30551a274fbb28275ba5c"}
Nov 02 13:06:23.545146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.545106    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2" event={"ID":"594df756-ef67-40a2-b480-4168ec6c7ac0","Type":"ContainerStarted","Data":"45da959819f02e827b732f7e6ac372dd9e6ae77a67889907effc978e87852410"}
Nov 02 13:06:23.590876 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.590799    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2" podStartSLOduration=1.590779944 podStartE2EDuration="1.590779944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:06:23.588097516 +0000 UTC m=+2329.694800364" watchObservedRunningTime="2026-11-02 13:06:23.590779944 +0000 UTC m=+2329.697482795"
Nov 02 13:06:23.718082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.718054    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:23.766456 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.766428    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-kube-api-access-48mcq\" (UniqueName: \"kubernetes.io/projected/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-kube-api-access-48mcq\") pod \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\" (UID: \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\") "
Nov 02 13:06:23.766629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.766484    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-xks-ca\") pod \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\" (UID: \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\") "
Nov 02 13:06:23.766629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.766520    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-openshift-ca\") pod \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\" (UID: \"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c\") "
Nov 02 13:06:23.766926 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.766904    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-openshift-ca" pod "7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c" (UID: "7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:23.767256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.767237    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:23.768717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.768694    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-xks-ca" pod "7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c" (UID: "7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:06:23.768852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.768836    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-kube-api-access-48mcq" pod "7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c" (UID: "7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c"). InnerVolumeSpecName "kube-api-access-48mcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:23.868368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.868344    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:23.868368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:23.868369    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48mcq\" (UniqueName: \"kubernetes.io/projected/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c-kube-api-access-48mcq\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:24.074587 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:24.074499    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:24.466409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:24.466383    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:06:24.548945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:24.548907    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"]
Nov 02 13:06:24.556896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:24.556855    5879 generic.go:356] "Generic (PLEG): container finished" podID="594df756-ef67-40a2-b480-4168ec6c7ac0" containerID="0e22335c6ad5d2714e07b4b8b4ce754593d556177ad30551a274fbb28275ba5c" exitCode=0
Nov 02 13:06:24.557117 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:24.556964    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2" event={"ID":"594df756-ef67-40a2-b480-4168ec6c7ac0","Type":"ContainerDied","Data":"0e22335c6ad5d2714e07b4b8b4ce754593d556177ad30551a274fbb28275ba5c"}
Nov 02 13:06:24.559173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:24.559071    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"
Nov 02 13:06:24.559173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:24.559141    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5" event={"ID":"7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c","Type":"ContainerDied","Data":"f58bf25263c0686b37993d15ce30bab83943f457fa488fa271b8e4f7d3cdf0ca"}
Nov 02 13:06:24.559602 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:24.559300    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f58bf25263c0686b37993d15ce30bab83943f457fa488fa271b8e4f7d3cdf0ca"
Nov 02 13:06:25.721542 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:25.721514    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:25.796813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:25.796781    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-openshift-ca\") pod \"594df756-ef67-40a2-b480-4168ec6c7ac0\" (UID: \"594df756-ef67-40a2-b480-4168ec6c7ac0\") "
Nov 02 13:06:25.797029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:25.796836    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-xks-ca\") pod \"594df756-ef67-40a2-b480-4168ec6c7ac0\" (UID: \"594df756-ef67-40a2-b480-4168ec6c7ac0\") "
Nov 02 13:06:25.797029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:25.796973    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594df756-ef67-40a2-b480-4168ec6c7ac0-kube-api-access-cnjz2\" (UniqueName: \"kubernetes.io/projected/594df756-ef67-40a2-b480-4168ec6c7ac0-kube-api-access-cnjz2\") pod \"594df756-ef67-40a2-b480-4168ec6c7ac0\" (UID: \"594df756-ef67-40a2-b480-4168ec6c7ac0\") "
Nov 02 13:06:25.797283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:25.797252    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-openshift-ca" pod "594df756-ef67-40a2-b480-4168ec6c7ac0" (UID: "594df756-ef67-40a2-b480-4168ec6c7ac0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:25.797781 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:25.797737    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:25.799311 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:25.799280    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-xks-ca" pod "594df756-ef67-40a2-b480-4168ec6c7ac0" (UID: "594df756-ef67-40a2-b480-4168ec6c7ac0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:06:25.799427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:25.799376    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594df756-ef67-40a2-b480-4168ec6c7ac0-kube-api-access-cnjz2" pod "594df756-ef67-40a2-b480-4168ec6c7ac0" (UID: "594df756-ef67-40a2-b480-4168ec6c7ac0"). InnerVolumeSpecName "kube-api-access-cnjz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:25.899245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:25.899217    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnjz2\" (UniqueName: \"kubernetes.io/projected/594df756-ef67-40a2-b480-4168ec6c7ac0-kube-api-access-cnjz2\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:25.899245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:25.899243    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594df756-ef67-40a2-b480-4168ec6c7ac0-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:26.000784 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:26.000753    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:26.577652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:26.577620    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"
Nov 02 13:06:26.577840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:26.577653    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2" event={"ID":"594df756-ef67-40a2-b480-4168ec6c7ac0","Type":"ContainerDied","Data":"45da959819f02e827b732f7e6ac372dd9e6ae77a67889907effc978e87852410"}
Nov 02 13:06:26.577840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:26.577709    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45da959819f02e827b732f7e6ac372dd9e6ae77a67889907effc978e87852410"
Nov 02 13:06:27.829899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:27.829860    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:29.549513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.549476    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj" podUID="31a9550d-09b4-47eb-b614-069f0f66186b" containerName="payload-processing" containerID="cri-o://eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266" gracePeriod=25
Nov 02 13:06:29.584751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.584728    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:29.823453 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.823429    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:06:29.950857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.950824    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-plugins-config-volume\") pod \"31a9550d-09b4-47eb-b614-069f0f66186b\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") "
Nov 02 13:06:29.951060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.950922    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-provider-credentials\") pod \"31a9550d-09b4-47eb-b614-069f0f66186b\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") "
Nov 02 13:06:29.951165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.951065    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-kube-api-access-b4h7t\" (UniqueName: \"kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-kube-api-access-b4h7t\") pod \"31a9550d-09b4-47eb-b614-069f0f66186b\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") "
Nov 02 13:06:29.951165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.951108    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-routing-overlay-volume\") pod \"31a9550d-09b4-47eb-b614-069f0f66186b\" (UID: \"31a9550d-09b4-47eb-b614-069f0f66186b\") "
Nov 02 13:06:29.951272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.951250    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-plugins-config-volume" pod "31a9550d-09b4-47eb-b614-069f0f66186b" (UID: "31a9550d-09b4-47eb-b614-069f0f66186b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:29.951520 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.951496    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-routing-overlay-volume" pod "31a9550d-09b4-47eb-b614-069f0f66186b" (UID: "31a9550d-09b4-47eb-b614-069f0f66186b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:29.951774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.951727    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:29.951774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.951745    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31a9550d-09b4-47eb-b614-069f0f66186b-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:29.953359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.953325    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-kube-api-access-b4h7t" pod "31a9550d-09b4-47eb-b614-069f0f66186b" (UID: "31a9550d-09b4-47eb-b614-069f0f66186b"). InnerVolumeSpecName "kube-api-access-b4h7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:29.953472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:29.953398    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-provider-credentials" pod "31a9550d-09b4-47eb-b614-069f0f66186b" (UID: "31a9550d-09b4-47eb-b614-069f0f66186b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:30.054216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.054152    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:30.054216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.054175    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4h7t\" (UniqueName: \"kubernetes.io/projected/31a9550d-09b4-47eb-b614-069f0f66186b-kube-api-access-b4h7t\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:30.612393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.612363    5879 generic.go:356] "Generic (PLEG): container finished" podID="31a9550d-09b4-47eb-b614-069f0f66186b" containerID="eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266" exitCode=0
Nov 02 13:06:30.612820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.612413    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"
Nov 02 13:06:30.612820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.612454    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj" event={"ID":"31a9550d-09b4-47eb-b614-069f0f66186b","Type":"ContainerDied","Data":"eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266"}
Nov 02 13:06:30.612820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.612512    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj" event={"ID":"31a9550d-09b4-47eb-b614-069f0f66186b","Type":"ContainerDied","Data":"a48b8e8a88e814702ac6d85368d08763253699bcd3fd1752c717647deeb3bbd0"}
Nov 02 13:06:30.612820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.612536    5879 scope.go:122] "RemoveContainer" containerID="eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266"
Nov 02 13:06:30.623224 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.623207    5879 scope.go:122] "RemoveContainer" containerID="eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266"
Nov 02 13:06:30.623505 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:06:30.623482    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266\": container with ID starting with eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266 not found: ID does not exist" containerID="eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266"
Nov 02 13:06:30.623581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.623518    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266"} err="failed to get container status \"eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266\": rpc error: code = NotFound desc = could not find container \"eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266\": container with ID starting with eaee7ff583406b61a8efa77c3ca8ce5f0903b45811b0d80613a9628b19ab2266 not found: ID does not exist"
Nov 02 13:06:30.670363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.670336    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"]
Nov 02 13:06:30.683688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.683663    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-7796g4kwj"]
Nov 02 13:06:30.802745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.802719    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"]
Nov 02 13:06:30.803437 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.803423    5879 cpu_manager.go:414] "containerMap: removing container" podUID="31a9550d-09b4-47eb-b614-069f0f66186b" containerName="payload-processing"
Nov 02 13:06:30.803488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.803438    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31a9550d-09b4-47eb-b614-069f0f66186b" containerName="payload-processing"
Nov 02 13:06:30.803488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.803450    5879 cpu_manager.go:414] "containerMap: removing container" podUID="7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c" containerName="revoke-keys"
Nov 02 13:06:30.803488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.803455    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c" containerName="revoke-keys"
Nov 02 13:06:30.803488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.803469    5879 cpu_manager.go:414] "containerMap: removing container" podUID="594df756-ef67-40a2-b480-4168ec6c7ac0" containerName="revoke-keys"
Nov 02 13:06:30.803488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.803476    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594df756-ef67-40a2-b480-4168ec6c7ac0" containerName="revoke-keys"
Nov 02 13:06:30.803662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.803601    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="594df756-ef67-40a2-b480-4168ec6c7ac0" containerName="revoke-keys"
Nov 02 13:06:30.803662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.803620    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="31a9550d-09b4-47eb-b614-069f0f66186b" containerName="payload-processing"
Nov 02 13:06:30.803662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.803627    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c" containerName="revoke-keys"
Nov 02 13:06:30.808563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.808548    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:30.824985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.824964    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:06:30.824985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.825035    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:06:30.827911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.827887    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"]
Nov 02 13:06:30.864283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.864255    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwglw\" (UniqueName: \"kubernetes.io/projected/d8916719-d328-48f3-b232-309806c64e7d-kube-api-access-lwglw\") pod \"maas-api-revoke-sub-222a3264-45fv9\" (UID: \"d8916719-d328-48f3-b232-309806c64e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:30.864422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.864394    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8916719-d328-48f3-b232-309806c64e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-222a3264-45fv9\" (UID: \"d8916719-d328-48f3-b232-309806c64e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:30.864484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.864472    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8916719-d328-48f3-b232-309806c64e7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-222a3264-45fv9\" (UID: \"d8916719-d328-48f3-b232-309806c64e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:30.966306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.966279    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8916719-d328-48f3-b232-309806c64e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-222a3264-45fv9\" (UID: \"d8916719-d328-48f3-b232-309806c64e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:30.966459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.966393    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8916719-d328-48f3-b232-309806c64e7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-222a3264-45fv9\" (UID: \"d8916719-d328-48f3-b232-309806c64e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:30.966518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.966502    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwglw\" (UniqueName: \"kubernetes.io/projected/d8916719-d328-48f3-b232-309806c64e7d-kube-api-access-lwglw\") pod \"maas-api-revoke-sub-222a3264-45fv9\" (UID: \"d8916719-d328-48f3-b232-309806c64e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:30.966984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.966960    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8916719-d328-48f3-b232-309806c64e7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-222a3264-45fv9\" (UID: \"d8916719-d328-48f3-b232-309806c64e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:30.968937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.968918    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8916719-d328-48f3-b232-309806c64e7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-222a3264-45fv9\" (UID: \"d8916719-d328-48f3-b232-309806c64e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:30.991518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:30.991494    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwglw\" (UniqueName: \"kubernetes.io/projected/d8916719-d328-48f3-b232-309806c64e7d-kube-api-access-lwglw\") pod \"maas-api-revoke-sub-222a3264-45fv9\" (UID: \"d8916719-d328-48f3-b232-309806c64e7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:31.123811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:31.123735    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:31.389968 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:31.389883    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:31.434890 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:06:31.433477    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8916719_d328_48f3_b232_309806c64e7d.slice/crio-3e1adcb47d5faebbafa027c510d037bd8945a6b306436bc682a5dd3039a48a5d WatchSource:0}: Error finding container 3e1adcb47d5faebbafa027c510d037bd8945a6b306436bc682a5dd3039a48a5d: Status 404 returned error can't find the container with id 3e1adcb47d5faebbafa027c510d037bd8945a6b306436bc682a5dd3039a48a5d
Nov 02 13:06:31.443843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:31.443812    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"]
Nov 02 13:06:31.620965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:31.620931    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9" event={"ID":"d8916719-d328-48f3-b232-309806c64e7d","Type":"ContainerStarted","Data":"3e1adcb47d5faebbafa027c510d037bd8945a6b306436bc682a5dd3039a48a5d"}
Nov 02 13:06:32.017897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:32.017823    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31a9550d-09b4-47eb-b614-069f0f66186b" path="/var/lib/kubelet/pods/31a9550d-09b4-47eb-b614-069f0f66186b/volumes"
Nov 02 13:06:32.631275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:32.631237    5879 generic.go:356] "Generic (PLEG): container finished" podID="d8916719-d328-48f3-b232-309806c64e7d" containerID="8dd99fc52a055cf31c6c51b5d23ef3197920a2450d45795036f07a46b452c41f" exitCode=0
Nov 02 13:06:32.631668 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:32.631317    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9" event={"ID":"d8916719-d328-48f3-b232-309806c64e7d","Type":"ContainerDied","Data":"8dd99fc52a055cf31c6c51b5d23ef3197920a2450d45795036f07a46b452c41f"}
Nov 02 13:06:33.217153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:33.217124    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:33.661762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:33.661730    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:33.671296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:33.671270    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"]
Nov 02 13:06:33.787951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:33.787924    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:33.814532 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:33.814508    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:33.903931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:33.903851    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8916719-d328-48f3-b232-309806c64e7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8916719-d328-48f3-b232-309806c64e7d-maas-api-xks-ca\") pod \"d8916719-d328-48f3-b232-309806c64e7d\" (UID: \"d8916719-d328-48f3-b232-309806c64e7d\") "
Nov 02 13:06:33.903931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:33.903893    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8916719-d328-48f3-b232-309806c64e7d-kube-api-access-lwglw\" (UniqueName: \"kubernetes.io/projected/d8916719-d328-48f3-b232-309806c64e7d-kube-api-access-lwglw\") pod \"d8916719-d328-48f3-b232-309806c64e7d\" (UID: \"d8916719-d328-48f3-b232-309806c64e7d\") "
Nov 02 13:06:33.904135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:33.904077    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8916719-d328-48f3-b232-309806c64e7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8916719-d328-48f3-b232-309806c64e7d-maas-api-openshift-ca\") pod \"d8916719-d328-48f3-b232-309806c64e7d\" (UID: \"d8916719-d328-48f3-b232-309806c64e7d\") "
Nov 02 13:06:33.904517 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:33.904488    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8916719-d328-48f3-b232-309806c64e7d-maas-api-openshift-ca" pod "d8916719-d328-48f3-b232-309806c64e7d" (UID: "d8916719-d328-48f3-b232-309806c64e7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:33.906251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:33.906225    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8916719-d328-48f3-b232-309806c64e7d-maas-api-xks-ca" pod "d8916719-d328-48f3-b232-309806c64e7d" (UID: "d8916719-d328-48f3-b232-309806c64e7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:06:33.906417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:33.906402    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8916719-d328-48f3-b232-309806c64e7d-kube-api-access-lwglw" pod "d8916719-d328-48f3-b232-309806c64e7d" (UID: "d8916719-d328-48f3-b232-309806c64e7d"). InnerVolumeSpecName "kube-api-access-lwglw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:34.008018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:34.006375    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8916719-d328-48f3-b232-309806c64e7d-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:34.008018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:34.006408    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwglw\" (UniqueName: \"kubernetes.io/projected/d8916719-d328-48f3-b232-309806c64e7d-kube-api-access-lwglw\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:34.008018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:34.006425    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8916719-d328-48f3-b232-309806c64e7d-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:34.269699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:34.269619    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:34.652478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:34.652452    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"
Nov 02 13:06:34.652692 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:34.652448    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9" event={"ID":"d8916719-d328-48f3-b232-309806c64e7d","Type":"ContainerDied","Data":"3e1adcb47d5faebbafa027c510d037bd8945a6b306436bc682a5dd3039a48a5d"}
Nov 02 13:06:34.652692 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:34.652612    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e1adcb47d5faebbafa027c510d037bd8945a6b306436bc682a5dd3039a48a5d"
Nov 02 13:06:35.185040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:35.185006    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:36.160780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:36.160747    5879 scope.go:122] "RemoveContainer" containerID="705ea9e08c8413ab96993364fa12399073e9ba1d928123807909c487586d825b"
Nov 02 13:06:36.171538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:36.171518    5879 scope.go:122] "RemoveContainer" containerID="b841218141918cc6aba1a0834881ab0ba160c6e728e36f4bb5952eff5d7942e7"
Nov 02 13:06:36.181218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:36.181197    5879 scope.go:122] "RemoveContainer" containerID="6a434892c71579d191319df7ea34550934da6d4381dc3e69a6b5f00a8442b1f0"
Nov 02 13:06:36.191907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:36.191877    5879 scope.go:122] "RemoveContainer" containerID="c75cc10bd782e79136db2849048795b5566bc126cbbc93dfac512ee867c043b7"
Nov 02 13:06:36.205086 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:36.205054    5879 scope.go:122] "RemoveContainer" containerID="77838c551780dcd6bf040fe365133f22d984f2c04f783b07c6f6c3c91cc36ddc"
Nov 02 13:06:36.222788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:36.222747    5879 scope.go:122] "RemoveContainer" containerID="00ab4dbbb2edb0b4758b7184036abd3f7f830d90e68281c1c1f140c10d90a008"
Nov 02 13:06:36.982923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:36.982889    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:37.974477 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:37.974433    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:38.672245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:38.672175    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc" podUID="2824542d-91bc-4809-b701-3005bf78f2ae" containerName="payload-processing" containerID="cri-o://bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9" gracePeriod=25
Nov 02 13:06:38.721430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:38.721407    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:38.957779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:38.957752    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:06:39.078293 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.078251    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-provider-credentials\") pod \"2824542d-91bc-4809-b701-3005bf78f2ae\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") "
Nov 02 13:06:39.078722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.078340    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-routing-overlay-volume\") pod \"2824542d-91bc-4809-b701-3005bf78f2ae\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") "
Nov 02 13:06:39.078722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.078486    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-kube-api-access-n7x8r\" (UniqueName: \"kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-kube-api-access-n7x8r\") pod \"2824542d-91bc-4809-b701-3005bf78f2ae\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") "
Nov 02 13:06:39.078722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.078556    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-plugins-config-volume\") pod \"2824542d-91bc-4809-b701-3005bf78f2ae\" (UID: \"2824542d-91bc-4809-b701-3005bf78f2ae\") "
Nov 02 13:06:39.078722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.078697    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-routing-overlay-volume" pod "2824542d-91bc-4809-b701-3005bf78f2ae" (UID: "2824542d-91bc-4809-b701-3005bf78f2ae"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:39.079033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.078984    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-plugins-config-volume" pod "2824542d-91bc-4809-b701-3005bf78f2ae" (UID: "2824542d-91bc-4809-b701-3005bf78f2ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:39.079144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.079127    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:39.079207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.079150    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2824542d-91bc-4809-b701-3005bf78f2ae-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:39.080653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.080634    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-kube-api-access-n7x8r" pod "2824542d-91bc-4809-b701-3005bf78f2ae" (UID: "2824542d-91bc-4809-b701-3005bf78f2ae"). InnerVolumeSpecName "kube-api-access-n7x8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:39.080729 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.080695    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-provider-credentials" pod "2824542d-91bc-4809-b701-3005bf78f2ae" (UID: "2824542d-91bc-4809-b701-3005bf78f2ae"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:39.181065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.180968    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7x8r\" (UniqueName: \"kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-kube-api-access-n7x8r\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:39.181065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.181030    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2824542d-91bc-4809-b701-3005bf78f2ae-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:39.690859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.690825    5879 generic.go:356] "Generic (PLEG): container finished" podID="2824542d-91bc-4809-b701-3005bf78f2ae" containerID="bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9" exitCode=0
Nov 02 13:06:39.691082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.690898    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"
Nov 02 13:06:39.691082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.690916    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc" event={"ID":"2824542d-91bc-4809-b701-3005bf78f2ae","Type":"ContainerDied","Data":"bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9"}
Nov 02 13:06:39.691082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.691060    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc" event={"ID":"2824542d-91bc-4809-b701-3005bf78f2ae","Type":"ContainerDied","Data":"0970a135c52db89d09dd4572cfababf29fb8326f52e047345e1781e2f1527b41"}
Nov 02 13:06:39.691273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.691089    5879 scope.go:122] "RemoveContainer" containerID="bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9"
Nov 02 13:06:39.702318 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.702301    5879 scope.go:122] "RemoveContainer" containerID="bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9"
Nov 02 13:06:39.702581 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:06:39.702560    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9\": container with ID starting with bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9 not found: ID does not exist" containerID="bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9"
Nov 02 13:06:39.702625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.702590    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9"} err="failed to get container status \"bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9\": rpc error: code = NotFound desc = could not find container \"bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9\": container with ID starting with bc0d2fbd655dc48d326f5bb163cab0c8425e72993743f123086696be30c01eb9 not found: ID does not exist"
Nov 02 13:06:39.749173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.749139    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"]
Nov 02 13:06:39.762193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:39.762163    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-936529/payload-processing-external-model-e2e-extmod-b-936529-74d8c9kmc"]
Nov 02 13:06:40.018868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:40.018788    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2824542d-91bc-4809-b701-3005bf78f2ae" path="/var/lib/kubelet/pods/2824542d-91bc-4809-b701-3005bf78f2ae/volumes"
Nov 02 13:06:40.478784 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:40.478754    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:41.076677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:41.076642    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:41.236320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:41.236290    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:41.363370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:41.363285    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:42.269325 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:42.269282    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:43.380163 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.380126    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"]
Nov 02 13:06:43.380976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.380959    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2824542d-91bc-4809-b701-3005bf78f2ae" containerName="payload-processing"
Nov 02 13:06:43.380976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.380977    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2824542d-91bc-4809-b701-3005bf78f2ae" containerName="payload-processing"
Nov 02 13:06:43.381118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.381012    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d8916719-d328-48f3-b232-309806c64e7d" containerName="revoke-keys"
Nov 02 13:06:43.381118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.381018    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8916719-d328-48f3-b232-309806c64e7d" containerName="revoke-keys"
Nov 02 13:06:43.381221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.381153    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2824542d-91bc-4809-b701-3005bf78f2ae" containerName="payload-processing"
Nov 02 13:06:43.381221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.381168    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8916719-d328-48f3-b232-309806c64e7d" containerName="revoke-keys"
Nov 02 13:06:43.386712 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.386689    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:43.403462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.403427    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"]
Nov 02 13:06:43.413138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.413109    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:06:43.413317 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.413172    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:06:43.441146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.441066    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353ac4be-xv8pc\" (UID: \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:43.441315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.441190    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2275j\" (UniqueName: \"kubernetes.io/projected/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-kube-api-access-2275j\") pod \"maas-api-revoke-sub-353ac4be-xv8pc\" (UID: \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:43.441378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.441354    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353ac4be-xv8pc\" (UID: \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:43.542394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.542355    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353ac4be-xv8pc\" (UID: \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:43.542619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.542440    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353ac4be-xv8pc\" (UID: \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:43.542619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.542568    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2275j\" (UniqueName: \"kubernetes.io/projected/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-kube-api-access-2275j\") pod \"maas-api-revoke-sub-353ac4be-xv8pc\" (UID: \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:43.543420 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.543396    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353ac4be-xv8pc\" (UID: \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:43.545114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.545093    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353ac4be-xv8pc\" (UID: \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:43.567651 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.567625    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2275j\" (UniqueName: \"kubernetes.io/projected/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-kube-api-access-2275j\") pod \"maas-api-revoke-sub-353ac4be-xv8pc\" (UID: \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:43.704269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:43.704180    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:44.010090 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:06:44.010058    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1062baa0_a9ae_4b91_be5e_4c7d39e001bc.slice/crio-4d1663b676a17bd880f6ca304d17f85db73a9e6679eb87d51fb64618ac252063 WatchSource:0}: Error finding container 4d1663b676a17bd880f6ca304d17f85db73a9e6679eb87d51fb64618ac252063: Status 404 returned error can't find the container with id 4d1663b676a17bd880f6ca304d17f85db73a9e6679eb87d51fb64618ac252063
Nov 02 13:06:44.022411 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:44.022377    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"]
Nov 02 13:06:44.053167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:44.053142    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:44.730977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:44.730944    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc" event={"ID":"1062baa0-a9ae-4b91-be5e-4c7d39e001bc","Type":"ContainerStarted","Data":"a7efb6b87f838b30e28d5be0e14bea184458c846933b49dc3a1144b750ed1df4"}
Nov 02 13:06:44.730977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:44.731013    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc" event={"ID":"1062baa0-a9ae-4b91-be5e-4c7d39e001bc","Type":"ContainerStarted","Data":"4d1663b676a17bd880f6ca304d17f85db73a9e6679eb87d51fb64618ac252063"}
Nov 02 13:06:44.775406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:44.775363    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc" podStartSLOduration=1.775349169 podStartE2EDuration="1.775349169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:06:44.773494069 +0000 UTC m=+2350.880196933" watchObservedRunningTime="2026-11-02 13:06:44.775349169 +0000 UTC m=+2350.882052021"
Nov 02 13:06:44.823823 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:06:44.823784    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:06:45.741726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.741693    5879 generic.go:356] "Generic (PLEG): container finished" podID="1062baa0-a9ae-4b91-be5e-4c7d39e001bc" containerID="a7efb6b87f838b30e28d5be0e14bea184458c846933b49dc3a1144b750ed1df4" exitCode=0
Nov 02 13:06:45.742156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.741788    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc" event={"ID":"1062baa0-a9ae-4b91-be5e-4c7d39e001bc","Type":"ContainerDied","Data":"a7efb6b87f838b30e28d5be0e14bea184458c846933b49dc3a1144b750ed1df4"}
Nov 02 13:06:45.746866 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.746840    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"]
Nov 02 13:06:45.751304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.751286    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:45.772190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.772158    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2mns\" (UniqueName: \"kubernetes.io/projected/a76c829c-3682-4e2b-b53b-8a1f35260090-kube-api-access-p2mns\") pod \"maas-api-revoke-sub-af805d35-n66tk\" (UID: \"a76c829c-3682-4e2b-b53b-8a1f35260090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:45.772366 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.772292    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af805d35-n66tk\" (UID: \"a76c829c-3682-4e2b-b53b-8a1f35260090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:45.772479 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.772456    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af805d35-n66tk\" (UID: \"a76c829c-3682-4e2b-b53b-8a1f35260090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:45.775315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.775294    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"]
Nov 02 13:06:45.777662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.777634    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:45.874133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.874101    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mns\" (UniqueName: \"kubernetes.io/projected/a76c829c-3682-4e2b-b53b-8a1f35260090-kube-api-access-p2mns\") pod \"maas-api-revoke-sub-af805d35-n66tk\" (UID: \"a76c829c-3682-4e2b-b53b-8a1f35260090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:45.874311 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.874187    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af805d35-n66tk\" (UID: \"a76c829c-3682-4e2b-b53b-8a1f35260090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:45.874311 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.874274    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af805d35-n66tk\" (UID: \"a76c829c-3682-4e2b-b53b-8a1f35260090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:45.874851 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.874833    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af805d35-n66tk\" (UID: \"a76c829c-3682-4e2b-b53b-8a1f35260090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:45.876699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.876681    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af805d35-n66tk\" (UID: \"a76c829c-3682-4e2b-b53b-8a1f35260090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:45.899345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:45.899319    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mns\" (UniqueName: \"kubernetes.io/projected/a76c829c-3682-4e2b-b53b-8a1f35260090-kube-api-access-p2mns\") pod \"maas-api-revoke-sub-af805d35-n66tk\" (UID: \"a76c829c-3682-4e2b-b53b-8a1f35260090\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:46.068121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.068038    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:46.374828 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:06:46.374796    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76c829c_3682_4e2b_b53b_8a1f35260090.slice/crio-ffe0e3e02dd6594f487487b63a0c882da0b5478d8d8dac503480b2a6903d9174 WatchSource:0}: Error finding container ffe0e3e02dd6594f487487b63a0c882da0b5478d8d8dac503480b2a6903d9174: Status 404 returned error can't find the container with id ffe0e3e02dd6594f487487b63a0c882da0b5478d8d8dac503480b2a6903d9174
Nov 02 13:06:46.375436 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.375413    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"]
Nov 02 13:06:46.750738 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.750695    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk" event={"ID":"a76c829c-3682-4e2b-b53b-8a1f35260090","Type":"ContainerStarted","Data":"d28db5c1a1d9a22c1582778200367191532fb8e796c4c225820b45f0fa02a274"}
Nov 02 13:06:46.751242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.750753    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk" event={"ID":"a76c829c-3682-4e2b-b53b-8a1f35260090","Type":"ContainerStarted","Data":"ffe0e3e02dd6594f487487b63a0c882da0b5478d8d8dac503480b2a6903d9174"}
Nov 02 13:06:46.798094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.798040    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk" podStartSLOduration=1.798021115 podStartE2EDuration="1.798021115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:06:46.794745452 +0000 UTC m=+2352.901448302" watchObservedRunningTime="2026-11-02 13:06:46.798021115 +0000 UTC m=+2352.904723967"
Nov 02 13:06:46.918586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.918557    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:46.991322 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.991237    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-openshift-ca\") pod \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\" (UID: \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\") "
Nov 02 13:06:46.991499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.991353    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-kube-api-access-2275j\" (UniqueName: \"kubernetes.io/projected/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-kube-api-access-2275j\") pod \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\" (UID: \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\") "
Nov 02 13:06:46.991499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.991479    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-xks-ca\") pod \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\" (UID: \"1062baa0-a9ae-4b91-be5e-4c7d39e001bc\") "
Nov 02 13:06:46.991675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.991652    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-openshift-ca" pod "1062baa0-a9ae-4b91-be5e-4c7d39e001bc" (UID: "1062baa0-a9ae-4b91-be5e-4c7d39e001bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:46.992158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.992134    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:46.993807 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.993776    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-kube-api-access-2275j" pod "1062baa0-a9ae-4b91-be5e-4c7d39e001bc" (UID: "1062baa0-a9ae-4b91-be5e-4c7d39e001bc"). InnerVolumeSpecName "kube-api-access-2275j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:46.994263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:46.994236    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-xks-ca" pod "1062baa0-a9ae-4b91-be5e-4c7d39e001bc" (UID: "1062baa0-a9ae-4b91-be5e-4c7d39e001bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:06:47.093632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:47.093601    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:47.093632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:47.093629    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2275j\" (UniqueName: \"kubernetes.io/projected/1062baa0-a9ae-4b91-be5e-4c7d39e001bc-kube-api-access-2275j\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:47.707279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:47.707176    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:47.759465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:47.759433    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"
Nov 02 13:06:47.759897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:47.759472    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc" event={"ID":"1062baa0-a9ae-4b91-be5e-4c7d39e001bc","Type":"ContainerDied","Data":"4d1663b676a17bd880f6ca304d17f85db73a9e6679eb87d51fb64618ac252063"}
Nov 02 13:06:47.759897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:47.759524    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d1663b676a17bd880f6ca304d17f85db73a9e6679eb87d51fb64618ac252063"
Nov 02 13:06:47.761323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:47.761297    5879 generic.go:356] "Generic (PLEG): container finished" podID="a76c829c-3682-4e2b-b53b-8a1f35260090" containerID="d28db5c1a1d9a22c1582778200367191532fb8e796c4c225820b45f0fa02a274" exitCode=0
Nov 02 13:06:47.761455 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:47.761381    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk" event={"ID":"a76c829c-3682-4e2b-b53b-8a1f35260090","Type":"ContainerDied","Data":"d28db5c1a1d9a22c1582778200367191532fb8e796c4c225820b45f0fa02a274"}
Nov 02 13:06:47.839806 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:47.839775    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:47.978823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:47.978725    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:48.101508 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:48.101466    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:48.273548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:48.273462    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:48.408853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:48.408815    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:48.545356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:48.545269    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:48.718776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:48.718742    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:48.946043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:48.946019    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:49.015794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.015756    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76c829c-3682-4e2b-b53b-8a1f35260090-kube-api-access-p2mns\" (UniqueName: \"kubernetes.io/projected/a76c829c-3682-4e2b-b53b-8a1f35260090-kube-api-access-p2mns\") pod \"a76c829c-3682-4e2b-b53b-8a1f35260090\" (UID: \"a76c829c-3682-4e2b-b53b-8a1f35260090\") "
Nov 02 13:06:49.016008 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.015941    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-xks-ca\") pod \"a76c829c-3682-4e2b-b53b-8a1f35260090\" (UID: \"a76c829c-3682-4e2b-b53b-8a1f35260090\") "
Nov 02 13:06:49.016167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.016146    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-openshift-ca\") pod \"a76c829c-3682-4e2b-b53b-8a1f35260090\" (UID: \"a76c829c-3682-4e2b-b53b-8a1f35260090\") "
Nov 02 13:06:49.016617 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.016585    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-openshift-ca" pod "a76c829c-3682-4e2b-b53b-8a1f35260090" (UID: "a76c829c-3682-4e2b-b53b-8a1f35260090"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:06:49.016782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.016741    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:49.019501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.019475    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-xks-ca" pod "a76c829c-3682-4e2b-b53b-8a1f35260090" (UID: "a76c829c-3682-4e2b-b53b-8a1f35260090"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:06:49.019620 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.019497    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76c829c-3682-4e2b-b53b-8a1f35260090-kube-api-access-p2mns" pod "a76c829c-3682-4e2b-b53b-8a1f35260090" (UID: "a76c829c-3682-4e2b-b53b-8a1f35260090"). InnerVolumeSpecName "kube-api-access-p2mns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:06:49.118312 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.118284    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2mns\" (UniqueName: \"kubernetes.io/projected/a76c829c-3682-4e2b-b53b-8a1f35260090-kube-api-access-p2mns\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:49.118312 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.118313    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a76c829c-3682-4e2b-b53b-8a1f35260090-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:06:49.343012 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.342965    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:49.475132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.475044    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:49.591539 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.591476    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:49.782733 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.782640    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"
Nov 02 13:06:49.782733 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.782668    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk" event={"ID":"a76c829c-3682-4e2b-b53b-8a1f35260090","Type":"ContainerDied","Data":"ffe0e3e02dd6594f487487b63a0c882da0b5478d8d8dac503480b2a6903d9174"}
Nov 02 13:06:49.782733 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:49.782722    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffe0e3e02dd6594f487487b63a0c882da0b5478d8d8dac503480b2a6903d9174"
Nov 02 13:06:50.463750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:50.463714    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:50.587365 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:50.587307    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:50.708053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:50.707978    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:50.977829 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:50.977792    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"]
Nov 02 13:06:50.978953 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:50.978933    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1062baa0-a9ae-4b91-be5e-4c7d39e001bc" containerName="revoke-keys"
Nov 02 13:06:50.979084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:50.978957    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1062baa0-a9ae-4b91-be5e-4c7d39e001bc" containerName="revoke-keys"
Nov 02 13:06:50.979084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:50.979034    5879 cpu_manager.go:414] "containerMap: removing container" podUID="a76c829c-3682-4e2b-b53b-8a1f35260090" containerName="revoke-keys"
Nov 02 13:06:50.979084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:50.979044    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76c829c-3682-4e2b-b53b-8a1f35260090" containerName="revoke-keys"
Nov 02 13:06:50.979228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:50.979214    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1062baa0-a9ae-4b91-be5e-4c7d39e001bc" containerName="revoke-keys"
Nov 02 13:06:50.979265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:50.979237    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76c829c-3682-4e2b-b53b-8a1f35260090" containerName="revoke-keys"
Nov 02 13:06:50.983358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:50.983335    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:50.999808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:50.999782    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"agc-cred-7577f881eb5f\""
Nov 02 13:06:51.003690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.003656    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"]
Nov 02 13:06:51.012608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.012559    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"openshift-service-ca.crt\""
Nov 02 13:06:51.012811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.012724    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"payload-processing-external-model-e2e--00beae95-dockercfg-t2lnb\""
Nov 02 13:06:51.013155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.013132    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"payload-processing-external-model-plugins-e2e-extmod-a-ef5efe\""
Nov 02 13:06:51.014398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.014375    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"routing-overlay\""
Nov 02 13:06:51.014618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.014600    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"kube-root-ca.crt\""
Nov 02 13:06:51.048132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.048088    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.048355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.048174    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfkdc\" (UniqueName: \"kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-kube-api-access-wfkdc\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.048355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.048233    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.048355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.048291    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.149756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.149719    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.149970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.149777    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfkdc\" (UniqueName: \"kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-kube-api-access-wfkdc\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.149970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.149810    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.149970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.149881    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.150486 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.150460    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.150486 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.150481    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.152473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.152447    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.174857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.174825    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfkdc\" (UniqueName: \"kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-kube-api-access-wfkdc\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.298666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.298580    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.565289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.565163    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:51.611142 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:06:51.611108    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07afebe0_ba34_4c12_8f58_4cfdff585d7f.slice/crio-2d8f1efca8bb153e9096d2233c286670675e1f6ad0b144d58832513a7ca55ca6 WatchSource:0}: Error finding container 2d8f1efca8bb153e9096d2233c286670675e1f6ad0b144d58832513a7ca55ca6: Status 404 returned error can't find the container with id 2d8f1efca8bb153e9096d2233c286670675e1f6ad0b144d58832513a7ca55ca6
Nov 02 13:06:51.611564 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.611542    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"]
Nov 02 13:06:51.785114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.785076    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"]
Nov 02 13:06:51.790361 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.790337    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:51.807303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.807277    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"agc-cred-463a4b859316\""
Nov 02 13:06:51.810004 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.809963    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"]
Nov 02 13:06:51.828934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.828845    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc" event={"ID":"07afebe0-ba34-4c12-8f58-4cfdff585d7f","Type":"ContainerStarted","Data":"94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94"}
Nov 02 13:06:51.828934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.828904    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc" event={"ID":"07afebe0-ba34-4c12-8f58-4cfdff585d7f","Type":"ContainerStarted","Data":"2d8f1efca8bb153e9096d2233c286670675e1f6ad0b144d58832513a7ca55ca6"}
Nov 02 13:06:51.829161 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.828937    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:06:51.868823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.868791    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:51.868823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.868825    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42df\" (UniqueName: \"kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-kube-api-access-x42df\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:51.869147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.869118    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:51.869280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.869265    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:51.878531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.878487    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc" podStartSLOduration=1.8784731140000002 podStartE2EDuration="1.878473114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:06:51.877704403 +0000 UTC m=+2357.984407255" watchObservedRunningTime="2026-11-02 13:06:51.878473114 +0000 UTC m=+2357.985175966"
Nov 02 13:06:51.971094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.971061    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:51.971254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.971173    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:51.971298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.971274    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:51.971369 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.971303    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42df\" (UniqueName: \"kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-kube-api-access-x42df\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:51.972116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.972086    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:51.972289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.972211    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:51.974517 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.974498    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:51.998955 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:51.998923    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42df\" (UniqueName: \"kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-kube-api-access-x42df\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:52.105170 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:52.105086    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:52.422329 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:06:52.422292    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcea38918_23c7_4e73_bb02_47c268e5305a.slice/crio-925aa465016cee6f28c7be2aa77f335bbed33616878e1c06c0a1e76d4bed0fcb WatchSource:0}: Error finding container 925aa465016cee6f28c7be2aa77f335bbed33616878e1c06c0a1e76d4bed0fcb: Status 404 returned error can't find the container with id 925aa465016cee6f28c7be2aa77f335bbed33616878e1c06c0a1e76d4bed0fcb
Nov 02 13:06:52.425131 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:52.425104    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"]
Nov 02 13:06:52.840797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:52.840697    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96" event={"ID":"cea38918-23c7-4e73-bb02-47c268e5305a","Type":"ContainerStarted","Data":"ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425"}
Nov 02 13:06:52.840797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:52.840758    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96" event={"ID":"cea38918-23c7-4e73-bb02-47c268e5305a","Type":"ContainerStarted","Data":"925aa465016cee6f28c7be2aa77f335bbed33616878e1c06c0a1e76d4bed0fcb"}
Nov 02 13:06:52.841415 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:52.840798    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:06:52.885085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:52.885014    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96" podStartSLOduration=1.8849778160000001 podStartE2EDuration="1.884977816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:06:52.884279065 +0000 UTC m=+2358.990981927" watchObservedRunningTime="2026-11-02 13:06:52.884977816 +0000 UTC m=+2358.991680665"
Nov 02 13:06:53.366854 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:53.366816    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:54.699158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:54.699113    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"]
Nov 02 13:06:54.737578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:54.737535    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:54.861107 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:54.861075    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:55.182522 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:55.182460    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:55.335118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:55.335076    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:56.944518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:56.944481    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"]
Nov 02 13:06:56.947171 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:56.947145    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:57.076211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:57.076173    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:57.350189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:57.350107    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:57.756592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:57.756560    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:57.906579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:57.906543    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:58.061357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:58.061272    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:58.205656 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:58.205613    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:58.338772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:58.338665    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:58.517754 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:58.517723    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:58.877441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:58.877407    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:59.216468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:59.216367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:59.338847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:59.338813    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:06:59.700031 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:59.699973    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd" podUID="e48bffff-84c9-4c96-a141-4bb96945d21d" containerName="payload-processing" containerID="cri-o://d63a426b52678e4684db7c1b1b48a54fc3514a8572572c3eda17a0cda7ff6378" gracePeriod=25
Nov 02 13:06:59.901280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:59.901247    5879 generic.go:356] "Generic (PLEG): container finished" podID="e48bffff-84c9-4c96-a141-4bb96945d21d" containerID="d63a426b52678e4684db7c1b1b48a54fc3514a8572572c3eda17a0cda7ff6378" exitCode=0
Nov 02 13:06:59.901495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:06:59.901325    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd" event={"ID":"e48bffff-84c9-4c96-a141-4bb96945d21d","Type":"ContainerDied","Data":"d63a426b52678e4684db7c1b1b48a54fc3514a8572572c3eda17a0cda7ff6378"}
Nov 02 13:07:00.003666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.003634    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:07:00.094465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.094425    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-routing-overlay-volume\") pod \"e48bffff-84c9-4c96-a141-4bb96945d21d\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") "
Nov 02 13:07:00.094664 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.094578    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-provider-credentials\") pod \"e48bffff-84c9-4c96-a141-4bb96945d21d\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") "
Nov 02 13:07:00.094757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.094741    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-kube-api-access-zsz4w\" (UniqueName: \"kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-kube-api-access-zsz4w\") pod \"e48bffff-84c9-4c96-a141-4bb96945d21d\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") "
Nov 02 13:07:00.094843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.094829    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-plugins-config-volume\") pod \"e48bffff-84c9-4c96-a141-4bb96945d21d\" (UID: \"e48bffff-84c9-4c96-a141-4bb96945d21d\") "
Nov 02 13:07:00.094902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.094868    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-routing-overlay-volume" pod "e48bffff-84c9-4c96-a141-4bb96945d21d" (UID: "e48bffff-84c9-4c96-a141-4bb96945d21d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:00.095361 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.095331    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-plugins-config-volume" pod "e48bffff-84c9-4c96-a141-4bb96945d21d" (UID: "e48bffff-84c9-4c96-a141-4bb96945d21d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:00.095523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.095456    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:00.095523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.095476    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e48bffff-84c9-4c96-a141-4bb96945d21d-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:00.097762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.097734    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-provider-credentials" pod "e48bffff-84c9-4c96-a141-4bb96945d21d" (UID: "e48bffff-84c9-4c96-a141-4bb96945d21d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:00.097900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.097773    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-kube-api-access-zsz4w" pod "e48bffff-84c9-4c96-a141-4bb96945d21d" (UID: "e48bffff-84c9-4c96-a141-4bb96945d21d"). InnerVolumeSpecName "kube-api-access-zsz4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:00.197322 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.197294    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:00.197322 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.197318    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsz4w\" (UniqueName: \"kubernetes.io/projected/e48bffff-84c9-4c96-a141-4bb96945d21d-kube-api-access-zsz4w\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:00.619351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.619319    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:00.732208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.732169    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:00.858771 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.858734    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"]
Nov 02 13:07:00.859484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.859470    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e48bffff-84c9-4c96-a141-4bb96945d21d" containerName="payload-processing"
Nov 02 13:07:00.859540 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.859486    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e48bffff-84c9-4c96-a141-4bb96945d21d" containerName="payload-processing"
Nov 02 13:07:00.859644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.859635    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e48bffff-84c9-4c96-a141-4bb96945d21d" containerName="payload-processing"
Nov 02 13:07:00.863859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.863839    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:00.885046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.885021    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"]
Nov 02 13:07:00.894666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.894635    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25f7a6\"/\"agc-cred-616227780a0e\""
Nov 02 13:07:00.894666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.894635    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-25f7a6\"/\"payload-processing-external-model-e2e--edace3a4-dockercfg-bjz6d\""
Nov 02 13:07:00.916524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.916477    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd" event={"ID":"e48bffff-84c9-4c96-a141-4bb96945d21d","Type":"ContainerDied","Data":"e7b928e588c8f45e2a20b8917e4cb1353956f89250a1a0ec01431ea09ed4e639"}
Nov 02 13:07:00.916736 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.916554    5879 scope.go:122] "RemoveContainer" containerID="d63a426b52678e4684db7c1b1b48a54fc3514a8572572c3eda17a0cda7ff6378"
Nov 02 13:07:00.916736 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.916491    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"
Nov 02 13:07:00.975289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.975258    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"]
Nov 02 13:07:00.979232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:00.979208    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-f577tbckd"]
Nov 02 13:07:01.009573 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.009533    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.009744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.009682    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.009905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.009873    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77nhd\" (UniqueName: \"kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-kube-api-access-77nhd\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.010084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.010066    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.111749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.111708    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.111943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.111832    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.112043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.112012    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77nhd\" (UniqueName: \"kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-kube-api-access-77nhd\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.112509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.112479    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.112802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.112547    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.113456 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.113425    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.114656 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.114628    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.140089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.140011    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77nhd\" (UniqueName: \"kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-kube-api-access-77nhd\") pod \"payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") " pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.179146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.179105    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.491854 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:07:01.491824    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556a66bf_9cc7_49ec_b9b5_066c7976e718.slice/crio-21f70096e189a8f51fbee3520658b88c8112c12b201c0fc1cf3fd3f5047951e4 WatchSource:0}: Error finding container 21f70096e189a8f51fbee3520658b88c8112c12b201c0fc1cf3fd3f5047951e4: Status 404 returned error can't find the container with id 21f70096e189a8f51fbee3520658b88c8112c12b201c0fc1cf3fd3f5047951e4
Nov 02 13:07:01.493447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.493421    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"]
Nov 02 13:07:01.926468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.926430    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw" event={"ID":"556a66bf-9cc7-49ec-b9b5-066c7976e718","Type":"ContainerStarted","Data":"c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543"}
Nov 02 13:07:01.926928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.926487    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw" event={"ID":"556a66bf-9cc7-49ec-b9b5-066c7976e718","Type":"ContainerStarted","Data":"21f70096e189a8f51fbee3520658b88c8112c12b201c0fc1cf3fd3f5047951e4"}
Nov 02 13:07:01.926928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.926644    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:01.945666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.945625    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr" podUID="1f1fcba3-2a1c-45bd-8845-eba530ce578b" containerName="payload-processing" containerID="cri-o://c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950" gracePeriod=25
Nov 02 13:07:01.971494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:01.971445    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw" podStartSLOduration=1.971432313 podStartE2EDuration="1.971432313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:07:01.969285933 +0000 UTC m=+2368.075988797" watchObservedRunningTime="2026-11-02 13:07:01.971432313 +0000 UTC m=+2368.078135177"
Nov 02 13:07:02.018780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.018746    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e48bffff-84c9-4c96-a141-4bb96945d21d" path="/var/lib/kubelet/pods/e48bffff-84c9-4c96-a141-4bb96945d21d/volumes"
Nov 02 13:07:02.062838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.062807    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:02.184465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.184338    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:02.245927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.245900    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:07:02.315406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.315372    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:02.331384 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.331350    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-kube-api-access-lh5g5\" (UniqueName: \"kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-kube-api-access-lh5g5\") pod \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") "
Nov 02 13:07:02.331574 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.331487    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-routing-overlay-volume\") pod \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") "
Nov 02 13:07:02.331654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.331603    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-provider-credentials\") pod \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") "
Nov 02 13:07:02.331719 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.331708    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-plugins-config-volume\") pod \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\" (UID: \"1f1fcba3-2a1c-45bd-8845-eba530ce578b\") "
Nov 02 13:07:02.331925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.331896    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-routing-overlay-volume" pod "1f1fcba3-2a1c-45bd-8845-eba530ce578b" (UID: "1f1fcba3-2a1c-45bd-8845-eba530ce578b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:02.332287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.332256    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-plugins-config-volume" pod "1f1fcba3-2a1c-45bd-8845-eba530ce578b" (UID: "1f1fcba3-2a1c-45bd-8845-eba530ce578b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:02.332439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.332422    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:02.332483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.332446    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1fcba3-2a1c-45bd-8845-eba530ce578b-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:02.334661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.334636    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-kube-api-access-lh5g5" pod "1f1fcba3-2a1c-45bd-8845-eba530ce578b" (UID: "1f1fcba3-2a1c-45bd-8845-eba530ce578b"). InnerVolumeSpecName "kube-api-access-lh5g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:02.334780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.334684    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-provider-credentials" pod "1f1fcba3-2a1c-45bd-8845-eba530ce578b" (UID: "1f1fcba3-2a1c-45bd-8845-eba530ce578b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:02.433765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.433728    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:02.433765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.433757    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5g5\" (UniqueName: \"kubernetes.io/projected/1f1fcba3-2a1c-45bd-8845-eba530ce578b-kube-api-access-lh5g5\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:02.509831 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.509736    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:02.844242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.844166    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:07:02.935389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.935342    5879 generic.go:356] "Generic (PLEG): container finished" podID="1f1fcba3-2a1c-45bd-8845-eba530ce578b" containerID="c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950" exitCode=0
Nov 02 13:07:02.935877 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.935409    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"
Nov 02 13:07:02.935877 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.935454    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr" event={"ID":"1f1fcba3-2a1c-45bd-8845-eba530ce578b","Type":"ContainerDied","Data":"c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950"}
Nov 02 13:07:02.935877 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.935506    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr" event={"ID":"1f1fcba3-2a1c-45bd-8845-eba530ce578b","Type":"ContainerDied","Data":"02f77e6c2ebaf458a77f81385489bce74266a7790792bd43def4f4a0c9c79be8"}
Nov 02 13:07:02.935877 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.935540    5879 scope.go:122] "RemoveContainer" containerID="c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950"
Nov 02 13:07:02.950532 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.950495    5879 scope.go:122] "RemoveContainer" containerID="c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950"
Nov 02 13:07:02.955622 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:07:02.955577    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950\": container with ID starting with c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950 not found: ID does not exist" containerID="c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950"
Nov 02 13:07:02.955819 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.955627    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950"} err="failed to get container status \"c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950\": rpc error: code = NotFound desc = could not find container \"c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950\": container with ID starting with c3825c860ec6f9e6dc76001763f6aa32d8f12817fabaafa91c2de0e71cffb950 not found: ID does not exist"
Nov 02 13:07:02.998753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:02.998718    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"]
Nov 02 13:07:03.002172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:03.002145    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-f7fb886qr"]
Nov 02 13:07:03.852930 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:03.852892    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:07:03.933272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:03.933239    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"]
Nov 02 13:07:04.020013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:04.019958    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1fcba3-2a1c-45bd-8845-eba530ce578b" path="/var/lib/kubelet/pods/1f1fcba3-2a1c-45bd-8845-eba530ce578b/volumes"
Nov 02 13:07:04.278193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:04.278157    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:05.939189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:05.939150    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:07.757721 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:07.757690    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:08.933850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:08.933805    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc" podUID="07afebe0-ba34-4c12-8f58-4cfdff585d7f" containerName="payload-processing" containerID="cri-o://94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94" gracePeriod=25
Nov 02 13:07:09.231579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.231531    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:09.233863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.233825    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:07:09.343772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.343725    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-plugins-config-volume\") pod \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") "
Nov 02 13:07:09.343971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.343863    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-kube-api-access-wfkdc\" (UniqueName: \"kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-kube-api-access-wfkdc\") pod \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") "
Nov 02 13:07:09.344077 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.343976    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-routing-overlay-volume\") pod \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") "
Nov 02 13:07:09.344140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.344090    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-provider-credentials\") pod \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\" (UID: \"07afebe0-ba34-4c12-8f58-4cfdff585d7f\") "
Nov 02 13:07:09.344290 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.344267    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-plugins-config-volume" pod "07afebe0-ba34-4c12-8f58-4cfdff585d7f" (UID: "07afebe0-ba34-4c12-8f58-4cfdff585d7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:09.344383 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.344348    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-routing-overlay-volume" pod "07afebe0-ba34-4c12-8f58-4cfdff585d7f" (UID: "07afebe0-ba34-4c12-8f58-4cfdff585d7f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:09.344714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.344639    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:09.344714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.344664    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07afebe0-ba34-4c12-8f58-4cfdff585d7f-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:09.346463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.346434    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-provider-credentials" pod "07afebe0-ba34-4c12-8f58-4cfdff585d7f" (UID: "07afebe0-ba34-4c12-8f58-4cfdff585d7f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:09.346636 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.346609    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-kube-api-access-wfkdc" pod "07afebe0-ba34-4c12-8f58-4cfdff585d7f" (UID: "07afebe0-ba34-4c12-8f58-4cfdff585d7f"). InnerVolumeSpecName "kube-api-access-wfkdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:09.356919 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.356883    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:09.445916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.445820    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfkdc\" (UniqueName: \"kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-kube-api-access-wfkdc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:09.445916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.445857    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07afebe0-ba34-4c12-8f58-4cfdff585d7f-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:09.483973 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.483888    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:09.646145 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.646117    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:09.771861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.771780    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:09.905500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.905466    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:09.992811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.992777    5879 generic.go:356] "Generic (PLEG): container finished" podID="07afebe0-ba34-4c12-8f58-4cfdff585d7f" containerID="94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94" exitCode=0
Nov 02 13:07:09.993218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.993091    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc" event={"ID":"07afebe0-ba34-4c12-8f58-4cfdff585d7f","Type":"ContainerDied","Data":"94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94"}
Nov 02 13:07:09.993218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.993186    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc" event={"ID":"07afebe0-ba34-4c12-8f58-4cfdff585d7f","Type":"ContainerDied","Data":"2d8f1efca8bb153e9096d2233c286670675e1f6ad0b144d58832513a7ca55ca6"}
Nov 02 13:07:09.993318 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.993219    5879 scope.go:122] "RemoveContainer" containerID="94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94"
Nov 02 13:07:09.993388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:09.993373    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"
Nov 02 13:07:10.005958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:10.005931    5879 scope.go:122] "RemoveContainer" containerID="94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94"
Nov 02 13:07:10.006300 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:07:10.006277    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94\": container with ID starting with 94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94 not found: ID does not exist" containerID="94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94"
Nov 02 13:07:10.006372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:10.006311    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94"} err="failed to get container status \"94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94\": rpc error: code = NotFound desc = could not find container \"94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94\": container with ID starting with 94e6bd13cccbf056ddb0a405ed3859120d1fe5f87290651601e6cdf313f18b94 not found: ID does not exist"
Nov 02 13:07:10.051783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:10.051693    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"]
Nov 02 13:07:10.065303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:10.065270    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-59bb88grc"]
Nov 02 13:07:10.069116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:10.069092    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:10.795100 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:10.795060    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:10.950739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:10.950708    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:11.261207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:11.261174    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:11.394172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:11.394135    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:11.570706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:11.570626    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:11.799963 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:11.799927    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:11.923597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:11.923560    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:12.024463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.024423    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07afebe0-ba34-4c12-8f58-4cfdff585d7f" path="/var/lib/kubelet/pods/07afebe0-ba34-4c12-8f58-4cfdff585d7f/volumes"
Nov 02 13:07:12.639947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.639915    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:12.870743 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.870709    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"]
Nov 02 13:07:12.871598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.871576    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1f1fcba3-2a1c-45bd-8845-eba530ce578b" containerName="payload-processing"
Nov 02 13:07:12.871598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.871598    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1fcba3-2a1c-45bd-8845-eba530ce578b" containerName="payload-processing"
Nov 02 13:07:12.871737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.871614    5879 cpu_manager.go:414] "containerMap: removing container" podUID="07afebe0-ba34-4c12-8f58-4cfdff585d7f" containerName="payload-processing"
Nov 02 13:07:12.871737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.871619    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07afebe0-ba34-4c12-8f58-4cfdff585d7f" containerName="payload-processing"
Nov 02 13:07:12.871802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.871741    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1fcba3-2a1c-45bd-8845-eba530ce578b" containerName="payload-processing"
Nov 02 13:07:12.871802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.871749    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="07afebe0-ba34-4c12-8f58-4cfdff585d7f" containerName="payload-processing"
Nov 02 13:07:12.875316 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.875296    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:12.893718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.893655    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"routing-overlay\""
Nov 02 13:07:12.898141 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.898116    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"]
Nov 02 13:07:12.907557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.906632    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"openshift-service-ca.crt\""
Nov 02 13:07:12.907557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.906778    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"payload-processing-external-model-plugins-e2e-extmod-a-035dac\""
Nov 02 13:07:12.907557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.907243    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"agc-cred-40c571fc97c6\""
Nov 02 13:07:12.907557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.907307    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"payload-processing-external-model-e2e--9bdd4eed-dockercfg-xr7dk\""
Nov 02 13:07:12.907557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.907339    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"kube-root-ca.crt\""
Nov 02 13:07:12.940263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:12.940230    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:07:13.021020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.013906    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.021020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.014196    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8dsr\" (UniqueName: \"kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-kube-api-access-f8dsr\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.021020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.014431    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.021020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.015248    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.021020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.015909    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:13.116938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.116898    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8dsr\" (UniqueName: \"kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-kube-api-access-f8dsr\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.117245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.117215    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.117382 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.117312    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.117512 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.117491    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.118153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.118127    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.118662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.118636    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.120641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.120616    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.142318 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.142275    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8dsr\" (UniqueName: \"kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-kube-api-access-f8dsr\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.143183 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.143145    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:13.190271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.190174    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:13.547133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.546914    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"]
Nov 02 13:07:13.606753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.606718    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"]
Nov 02 13:07:13.611790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.611764    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.629586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.628981    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"agc-cred-66592af035d4\""
Nov 02 13:07:13.631699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.631675    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"]
Nov 02 13:07:13.752305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.752273    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.752756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.752447    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.752756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.752470    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgsv4\" (UniqueName: \"kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-kube-api-access-rgsv4\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.752756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.752596    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.854435 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.854358    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.854605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.854455    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.854668 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.854650    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.854711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.854688    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgsv4\" (UniqueName: \"kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-kube-api-access-rgsv4\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.855343 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.855316    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.855568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.855525    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.857279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.857249    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.884129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.884100    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgsv4\" (UniqueName: \"kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-kube-api-access-rgsv4\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.928473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.928437    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:13.945144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:13.945121    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:14.047020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.046964    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4" event={"ID":"987c7660-389a-4b5d-9e23-d9259a63131a","Type":"ContainerStarted","Data":"c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb"}
Nov 02 13:07:14.047217 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.047100    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4" event={"ID":"987c7660-389a-4b5d-9e23-d9259a63131a","Type":"ContainerStarted","Data":"93d6809a716e8a4e467e36f208a8f05b00f00ea508f1e9fcb53f85bdc25c65c8"}
Nov 02 13:07:14.047283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.047247    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:14.094465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.094400    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4" podStartSLOduration=2.094381526 podStartE2EDuration="2.094381526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:07:14.093278984 +0000 UTC m=+2380.199981834" watchObservedRunningTime="2026-11-02 13:07:14.094381526 +0000 UTC m=+2380.201084378"
Nov 02 13:07:14.171548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.171508    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:14.247302 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:07:14.244289    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d3caee_b5f3_4755_ab24_fef15f7f2fcd.slice/crio-2520f59cb8e20ac7e06753c4433f639c9fa53f565029b60cf30b4182cab714e1 WatchSource:0}: Error finding container 2520f59cb8e20ac7e06753c4433f639c9fa53f565029b60cf30b4182cab714e1: Status 404 returned error can't find the container with id 2520f59cb8e20ac7e06753c4433f639c9fa53f565029b60cf30b4182cab714e1
Nov 02 13:07:14.247302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.247228    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"]
Nov 02 13:07:14.326173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.326134    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:14.554742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.554648    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"]
Nov 02 13:07:14.560546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.560521    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.570851 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.570816    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.571050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.570887    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfqsz\" (UniqueName: \"kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-kube-api-access-mfqsz\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.571050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.570950    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.571173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.571079    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.577239 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.577202    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"agc-cred-8aa4cba0f301\""
Nov 02 13:07:14.577835 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.577805    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"]
Nov 02 13:07:14.671772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.671735    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.671986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.671806    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfqsz\" (UniqueName: \"kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-kube-api-access-mfqsz\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.671986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.671883    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.671986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.671981    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.673066 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.672987    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.673540 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.673472    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.675163 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.675134    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.697255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.697220    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfqsz\" (UniqueName: \"kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-kube-api-access-mfqsz\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:14.823098 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:07:14.822981    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:07:14.878105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:14.878071    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:15.056332 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.056295    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v" event={"ID":"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd","Type":"ContainerStarted","Data":"8ea45952d8f28ade9ddd0a88579d5347ef201ea61ccdf9114afa3d0a6c39c59f"}
Nov 02 13:07:15.056509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.056348    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v" event={"ID":"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd","Type":"ContainerStarted","Data":"2520f59cb8e20ac7e06753c4433f639c9fa53f565029b60cf30b4182cab714e1"}
Nov 02 13:07:15.056563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.056509    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:15.100566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.100468    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v" podStartSLOduration=2.100453919 podStartE2EDuration="2.100453919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:07:15.099660747 +0000 UTC m=+2381.206363598" watchObservedRunningTime="2026-11-02 13:07:15.100453919 +0000 UTC m=+2381.207156779"
Nov 02 13:07:15.187108 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:07:15.187077    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa28ea8c_32f7_4962_9528_dfd1c10dd44a.slice/crio-64f60c732c771370dfaa074d88c25ca5acad3acee5200f93a0d764c3b3990f51 WatchSource:0}: Error finding container 64f60c732c771370dfaa074d88c25ca5acad3acee5200f93a0d764c3b3990f51: Status 404 returned error can't find the container with id 64f60c732c771370dfaa074d88c25ca5acad3acee5200f93a0d764c3b3990f51
Nov 02 13:07:15.187788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.187762    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"]
Nov 02 13:07:15.515470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.515439    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"]
Nov 02 13:07:15.519948 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.519924    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.522251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.522216    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"]
Nov 02 13:07:15.536526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.536494    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"agc-cred-2276e6ef05cc\""
Nov 02 13:07:15.581909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.581860    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.581909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.581904    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m9c7\" (UniqueName: \"kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-kube-api-access-9m9c7\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.582242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.582104    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.582308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.582243    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.683081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.683046    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.683272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.683148    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.683272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.683201    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.683272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.683221    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m9c7\" (UniqueName: \"kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-kube-api-access-9m9c7\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.683915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.683887    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.684043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.683969    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.686036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.686015    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.708475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.708441    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m9c7\" (UniqueName: \"kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-kube-api-access-9m9c7\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.836452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.836370    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:15.865374 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:15.865331    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:16.066601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:16.066556    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt" event={"ID":"fa28ea8c-32f7-4962-9528-dfd1c10dd44a","Type":"ContainerStarted","Data":"8f350e5bf01112371bd2de757c21c243cacb8a8c43b369eec0d5cbde8ec921f8"}
Nov 02 13:07:16.066771 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:16.066621    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt" event={"ID":"fa28ea8c-32f7-4962-9528-dfd1c10dd44a","Type":"ContainerStarted","Data":"64f60c732c771370dfaa074d88c25ca5acad3acee5200f93a0d764c3b3990f51"}
Nov 02 13:07:16.066771 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:16.066664    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:16.117316 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:16.117263    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt" podStartSLOduration=2.117248789 podStartE2EDuration="2.117248789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:07:16.112454848 +0000 UTC m=+2382.219157702" watchObservedRunningTime="2026-11-02 13:07:16.117248789 +0000 UTC m=+2382.223951652"
Nov 02 13:07:16.160131 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:07:16.160096    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c13cde5_9860_4b39_ae37_c62d6ce8cca5.slice/crio-e9f4ffb86bdb7f85c35a4980dfad1cdca48724dd2dad108ff60ef28100688e2c WatchSource:0}: Error finding container e9f4ffb86bdb7f85c35a4980dfad1cdca48724dd2dad108ff60ef28100688e2c: Status 404 returned error can't find the container with id e9f4ffb86bdb7f85c35a4980dfad1cdca48724dd2dad108ff60ef28100688e2c
Nov 02 13:07:16.161966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:16.161939    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"]
Nov 02 13:07:17.079067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:17.079021    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c" event={"ID":"4c13cde5-9860-4b39-ae37-c62d6ce8cca5","Type":"ContainerStarted","Data":"cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8"}
Nov 02 13:07:17.079608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:17.079091    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c" event={"ID":"4c13cde5-9860-4b39-ae37-c62d6ce8cca5","Type":"ContainerStarted","Data":"e9f4ffb86bdb7f85c35a4980dfad1cdca48724dd2dad108ff60ef28100688e2c"}
Nov 02 13:07:17.079608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:17.079122    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:17.150310 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:17.150252    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c" podStartSLOduration=2.150230285 podStartE2EDuration="2.150230285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:07:17.148655644 +0000 UTC m=+2383.255358498" watchObservedRunningTime="2026-11-02 13:07:17.150230285 +0000 UTC m=+2383.256933198"
Nov 02 13:07:17.802657 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:17.802619    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:19.549189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:19.549148    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:21.288713 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:21.288676    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:22.975283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:22.975242    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:24.689128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:24.689095    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:25.059905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:25.059821    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:26.070204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:26.070176    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:07:26.150338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:26.150293    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"]
Nov 02 13:07:26.534152 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:26.534120    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:27.082683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:27.082657    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:07:27.161936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:27.161904    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"]
Nov 02 13:07:28.090189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:28.090158    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:07:28.169631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:28.169591    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"]
Nov 02 13:07:28.278681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:28.278650    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:29.985745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:29.985711    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:30.509478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:30.509448    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:30.644301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:30.644272    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:31.151004 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.150966    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw" podUID="5d729291-dafd-48a9-aa6d-1d8f06f0d729" containerName="payload-processing" containerID="cri-o://b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af" gracePeriod=25
Nov 02 13:07:31.440629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.440599    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:07:31.622940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.622903    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-plugins-config-volume\") pod \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") "
Nov 02 13:07:31.623169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.623135    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-routing-overlay-volume\") pod \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") "
Nov 02 13:07:31.623169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.623163    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-provider-credentials\") pod \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") "
Nov 02 13:07:31.623320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.623281    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-kube-api-access-bvt7c\" (UniqueName: \"kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-kube-api-access-bvt7c\") pod \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\" (UID: \"5d729291-dafd-48a9-aa6d-1d8f06f0d729\") "
Nov 02 13:07:31.623509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.623480    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-plugins-config-volume" pod "5d729291-dafd-48a9-aa6d-1d8f06f0d729" (UID: "5d729291-dafd-48a9-aa6d-1d8f06f0d729"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:31.623509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.623495    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-routing-overlay-volume" pod "5d729291-dafd-48a9-aa6d-1d8f06f0d729" (UID: "5d729291-dafd-48a9-aa6d-1d8f06f0d729"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:31.623847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.623828    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:31.623906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.623851    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d729291-dafd-48a9-aa6d-1d8f06f0d729-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:31.625654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.625631    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-kube-api-access-bvt7c" pod "5d729291-dafd-48a9-aa6d-1d8f06f0d729" (UID: "5d729291-dafd-48a9-aa6d-1d8f06f0d729"). InnerVolumeSpecName "kube-api-access-bvt7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:31.626078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.625730    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-provider-credentials" pod "5d729291-dafd-48a9-aa6d-1d8f06f0d729" (UID: "5d729291-dafd-48a9-aa6d-1d8f06f0d729"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:31.725581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.725500    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:31.725581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.725530    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvt7c\" (UniqueName: \"kubernetes.io/projected/5d729291-dafd-48a9-aa6d-1d8f06f0d729-kube-api-access-bvt7c\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:31.792605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:31.792571    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:32.163089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.163051    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4" podUID="987c7660-389a-4b5d-9e23-d9259a63131a" containerName="payload-processing" containerID="cri-o://c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb" gracePeriod=25
Nov 02 13:07:32.211235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.211207    5879 generic.go:356] "Generic (PLEG): container finished" podID="5d729291-dafd-48a9-aa6d-1d8f06f0d729" containerID="b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af" exitCode=0
Nov 02 13:07:32.211352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.211254    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"
Nov 02 13:07:32.211352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.211288    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw" event={"ID":"5d729291-dafd-48a9-aa6d-1d8f06f0d729","Type":"ContainerDied","Data":"b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af"}
Nov 02 13:07:32.211352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.211339    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw" event={"ID":"5d729291-dafd-48a9-aa6d-1d8f06f0d729","Type":"ContainerDied","Data":"a709604e38692596f45e25ec91ad98bfd05ab9ac62868d92fa76119f469358f4"}
Nov 02 13:07:32.211494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.211360    5879 scope.go:122] "RemoveContainer" containerID="b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af"
Nov 02 13:07:32.276783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.276763    5879 scope.go:122] "RemoveContainer" containerID="b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af"
Nov 02 13:07:32.277189 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:07:32.277164    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af\": container with ID starting with b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af not found: ID does not exist" containerID="b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af"
Nov 02 13:07:32.277269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.277195    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af"} err="failed to get container status \"b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af\": rpc error: code = NotFound desc = could not find container \"b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af\": container with ID starting with b0601ac55885d762501e4daba6f170cd0c9931456f22ec6b7387645333b2c8af not found: ID does not exist"
Nov 02 13:07:32.371017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.370971    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"]
Nov 02 13:07:32.388037 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.387737    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcmdgdw"]
Nov 02 13:07:32.450514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.450484    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:32.635811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.635783    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-plugins-config-volume\") pod \"987c7660-389a-4b5d-9e23-d9259a63131a\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") "
Nov 02 13:07:32.635984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.635935    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-kube-api-access-f8dsr\" (UniqueName: \"kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-kube-api-access-f8dsr\") pod \"987c7660-389a-4b5d-9e23-d9259a63131a\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") "
Nov 02 13:07:32.635984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.635969    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-provider-credentials\") pod \"987c7660-389a-4b5d-9e23-d9259a63131a\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") "
Nov 02 13:07:32.636107 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.636057    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-routing-overlay-volume\") pod \"987c7660-389a-4b5d-9e23-d9259a63131a\" (UID: \"987c7660-389a-4b5d-9e23-d9259a63131a\") "
Nov 02 13:07:32.636214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.636189    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-plugins-config-volume" pod "987c7660-389a-4b5d-9e23-d9259a63131a" (UID: "987c7660-389a-4b5d-9e23-d9259a63131a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:32.636441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.636418    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-routing-overlay-volume" pod "987c7660-389a-4b5d-9e23-d9259a63131a" (UID: "987c7660-389a-4b5d-9e23-d9259a63131a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:32.636586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.636543    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:32.636586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.636560    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/987c7660-389a-4b5d-9e23-d9259a63131a-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:32.638378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.638356    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-provider-credentials" pod "987c7660-389a-4b5d-9e23-d9259a63131a" (UID: "987c7660-389a-4b5d-9e23-d9259a63131a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:32.638476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.638388    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-kube-api-access-f8dsr" pod "987c7660-389a-4b5d-9e23-d9259a63131a" (UID: "987c7660-389a-4b5d-9e23-d9259a63131a"). InnerVolumeSpecName "kube-api-access-f8dsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:32.738352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.738318    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8dsr\" (UniqueName: \"kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-kube-api-access-f8dsr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:32.738352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:32.738344    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/987c7660-389a-4b5d-9e23-d9259a63131a-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:33.170553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.170511    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8" podUID="e0f9c526-0a62-43aa-a9c1-14efa35acbd8" containerName="payload-processing" containerID="cri-o://564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41" gracePeriod=25
Nov 02 13:07:33.220626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.220596    5879 generic.go:356] "Generic (PLEG): container finished" podID="987c7660-389a-4b5d-9e23-d9259a63131a" containerID="c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb" exitCode=0
Nov 02 13:07:33.220812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.220647    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"
Nov 02 13:07:33.220812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.220679    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4" event={"ID":"987c7660-389a-4b5d-9e23-d9259a63131a","Type":"ContainerDied","Data":"c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb"}
Nov 02 13:07:33.220812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.220740    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4" event={"ID":"987c7660-389a-4b5d-9e23-d9259a63131a","Type":"ContainerDied","Data":"93d6809a716e8a4e467e36f208a8f05b00f00ea508f1e9fcb53f85bdc25c65c8"}
Nov 02 13:07:33.220812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.220770    5879 scope.go:122] "RemoveContainer" containerID="c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb"
Nov 02 13:07:33.284190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.284161    5879 scope.go:122] "RemoveContainer" containerID="c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb"
Nov 02 13:07:33.284585 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:07:33.284562    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb\": container with ID starting with c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb not found: ID does not exist" containerID="c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb"
Nov 02 13:07:33.284689 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.284592    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb"} err="failed to get container status \"c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb\": rpc error: code = NotFound desc = could not find container \"c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb\": container with ID starting with c3f7283fd408b71438ee8406dc6d8661ea91222de5091bb4a54add511fd322bb not found: ID does not exist"
Nov 02 13:07:33.314469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.314435    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"]
Nov 02 13:07:33.327847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.327817    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccvgvn4"]
Nov 02 13:07:33.462087 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.462061    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:07:33.556078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.556043    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-kube-api-access-qnhtw\" (UniqueName: \"kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-kube-api-access-qnhtw\") pod \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") "
Nov 02 13:07:33.556247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.556145    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-routing-overlay-volume\") pod \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") "
Nov 02 13:07:33.556247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.556166    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-plugins-config-volume\") pod \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") "
Nov 02 13:07:33.556247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.556218    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-provider-credentials\") pod \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\" (UID: \"e0f9c526-0a62-43aa-a9c1-14efa35acbd8\") "
Nov 02 13:07:33.556613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.556585    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-routing-overlay-volume" pod "e0f9c526-0a62-43aa-a9c1-14efa35acbd8" (UID: "e0f9c526-0a62-43aa-a9c1-14efa35acbd8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:33.556613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.556605    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-plugins-config-volume" pod "e0f9c526-0a62-43aa-a9c1-14efa35acbd8" (UID: "e0f9c526-0a62-43aa-a9c1-14efa35acbd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:33.556856 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.556710    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:33.556856 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.556727    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:33.558522 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.558495    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-kube-api-access-qnhtw" pod "e0f9c526-0a62-43aa-a9c1-14efa35acbd8" (UID: "e0f9c526-0a62-43aa-a9c1-14efa35acbd8"). InnerVolumeSpecName "kube-api-access-qnhtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:33.558640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.558521    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-provider-credentials" pod "e0f9c526-0a62-43aa-a9c1-14efa35acbd8" (UID: "e0f9c526-0a62-43aa-a9c1-14efa35acbd8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:33.657810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.657770    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnhtw\" (UniqueName: \"kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-kube-api-access-qnhtw\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:33.657810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.657803    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0f9c526-0a62-43aa-a9c1-14efa35acbd8-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:33.869616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:33.869575    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:34.019462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:34.019425    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d729291-dafd-48a9-aa6d-1d8f06f0d729" path="/var/lib/kubelet/pods/5d729291-dafd-48a9-aa6d-1d8f06f0d729/volumes"
Nov 02 13:07:34.020059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:34.020038    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987c7660-389a-4b5d-9e23-d9259a63131a" path="/var/lib/kubelet/pods/987c7660-389a-4b5d-9e23-d9259a63131a/volumes"
Nov 02 13:07:34.231473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:34.231381    5879 generic.go:356] "Generic (PLEG): container finished" podID="e0f9c526-0a62-43aa-a9c1-14efa35acbd8" containerID="564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41" exitCode=0
Nov 02 13:07:34.231473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:34.231433    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"
Nov 02 13:07:34.232023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:34.231466    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8" event={"ID":"e0f9c526-0a62-43aa-a9c1-14efa35acbd8","Type":"ContainerDied","Data":"564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41"}
Nov 02 13:07:34.232023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:34.231527    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8" event={"ID":"e0f9c526-0a62-43aa-a9c1-14efa35acbd8","Type":"ContainerDied","Data":"81c7660980cf1030a9ca8a2b6a384a97d2e200e38d01e97e63e0015f1612aa3d"}
Nov 02 13:07:34.232023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:34.231554    5879 scope.go:122] "RemoveContainer" containerID="564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41"
Nov 02 13:07:34.242020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:34.241969    5879 scope.go:122] "RemoveContainer" containerID="564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41"
Nov 02 13:07:34.242309 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:07:34.242285    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41\": container with ID starting with 564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41 not found: ID does not exist" containerID="564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41"
Nov 02 13:07:34.242382 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:34.242318    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41"} err="failed to get container status \"564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41\": rpc error: code = NotFound desc = could not find container \"564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41\": container with ID starting with 564decb3ba74c8f159dc03f0b09e5592b1ca30e042f1eaeb86c8281bec89bb41 not found: ID does not exist"
Nov 02 13:07:34.300869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:34.300838    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"]
Nov 02 13:07:34.306393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:34.306359    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bczlxd8"]
Nov 02 13:07:35.706549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:35.706504    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:36.019195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:36.019103    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f9c526-0a62-43aa-a9c1-14efa35acbd8" path="/var/lib/kubelet/pods/e0f9c526-0a62-43aa-a9c1-14efa35acbd8/volumes"
Nov 02 13:07:37.482234 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:37.482198    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:38.208222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:38.208181    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:38.372936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:38.372903    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:39.299793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:39.299738    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:41.305721 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.305686    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"]
Nov 02 13:07:41.306494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.306478    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5d729291-dafd-48a9-aa6d-1d8f06f0d729" containerName="payload-processing"
Nov 02 13:07:41.306566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.306496    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d729291-dafd-48a9-aa6d-1d8f06f0d729" containerName="payload-processing"
Nov 02 13:07:41.306566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.306515    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e0f9c526-0a62-43aa-a9c1-14efa35acbd8" containerName="payload-processing"
Nov 02 13:07:41.306566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.306521    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f9c526-0a62-43aa-a9c1-14efa35acbd8" containerName="payload-processing"
Nov 02 13:07:41.306566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.306552    5879 cpu_manager.go:414] "containerMap: removing container" podUID="987c7660-389a-4b5d-9e23-d9259a63131a" containerName="payload-processing"
Nov 02 13:07:41.306566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.306557    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987c7660-389a-4b5d-9e23-d9259a63131a" containerName="payload-processing"
Nov 02 13:07:41.306756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.306668    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d729291-dafd-48a9-aa6d-1d8f06f0d729" containerName="payload-processing"
Nov 02 13:07:41.306756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.306678    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f9c526-0a62-43aa-a9c1-14efa35acbd8" containerName="payload-processing"
Nov 02 13:07:41.306756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.306689    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="987c7660-389a-4b5d-9e23-d9259a63131a" containerName="payload-processing"
Nov 02 13:07:41.312305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.312281    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:41.328717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.328689    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:07:41.330694 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.330668    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"]
Nov 02 13:07:41.339296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.339273    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:07:41.364680 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.364635    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c362144d-t9tpb\" (UID: \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:41.364858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.364743    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c362144d-t9tpb\" (UID: \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:41.365035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.365012    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qghps\" (UniqueName: \"kubernetes.io/projected/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-kube-api-access-qghps\") pod \"maas-api-revoke-sub-c362144d-t9tpb\" (UID: \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:41.436049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.436007    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:41.466937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.466894    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c362144d-t9tpb\" (UID: \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:41.467167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.466971    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c362144d-t9tpb\" (UID: \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:41.467167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.467143    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qghps\" (UniqueName: \"kubernetes.io/projected/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-kube-api-access-qghps\") pod \"maas-api-revoke-sub-c362144d-t9tpb\" (UID: \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:41.467768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.467748    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c362144d-t9tpb\" (UID: \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:41.469751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.469724    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c362144d-t9tpb\" (UID: \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:41.492912 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.492882    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qghps\" (UniqueName: \"kubernetes.io/projected/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-kube-api-access-qghps\") pod \"maas-api-revoke-sub-c362144d-t9tpb\" (UID: \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:41.628237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.628200    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:41.936269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:41.936236    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"]
Nov 02 13:07:41.936400 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:07:41.936242    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95442b0e_e7b1_403f_9ad3_a6121c7b6b95.slice/crio-99d82aba005ff49a95cde70ac78282a72597e0535ec96b57e520e366773c0728 WatchSource:0}: Error finding container 99d82aba005ff49a95cde70ac78282a72597e0535ec96b57e520e366773c0728: Status 404 returned error can't find the container with id 99d82aba005ff49a95cde70ac78282a72597e0535ec96b57e520e366773c0728
Nov 02 13:07:42.299815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:42.299784    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb" event={"ID":"95442b0e-e7b1-403f-9ad3-a6121c7b6b95","Type":"ContainerStarted","Data":"99d82aba005ff49a95cde70ac78282a72597e0535ec96b57e520e366773c0728"}
Nov 02 13:07:43.275544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:43.275464    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:43.310451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:43.310413    5879 generic.go:356] "Generic (PLEG): container finished" podID="95442b0e-e7b1-403f-9ad3-a6121c7b6b95" containerID="1f47390f1dd9f7e0f1ebdbb4c599de0b1aca24ebc27649570ed3ef828c9fe841" exitCode=0
Nov 02 13:07:43.310648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:43.310493    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb" event={"ID":"95442b0e-e7b1-403f-9ad3-a6121c7b6b95","Type":"ContainerDied","Data":"1f47390f1dd9f7e0f1ebdbb4c599de0b1aca24ebc27649570ed3ef828c9fe841"}
Nov 02 13:07:44.480559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:44.480534    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:44.610700 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:44.610600    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-kube-api-access-qghps\" (UniqueName: \"kubernetes.io/projected/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-kube-api-access-qghps\") pod \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\" (UID: \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\") "
Nov 02 13:07:44.610899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:44.610704    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-openshift-ca\") pod \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\" (UID: \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\") "
Nov 02 13:07:44.610899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:44.610829    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-xks-ca\") pod \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\" (UID: \"95442b0e-e7b1-403f-9ad3-a6121c7b6b95\") "
Nov 02 13:07:44.611173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:44.611149    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-openshift-ca" pod "95442b0e-e7b1-403f-9ad3-a6121c7b6b95" (UID: "95442b0e-e7b1-403f-9ad3-a6121c7b6b95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:44.611464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:44.611444    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:44.613132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:44.613103    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-kube-api-access-qghps" pod "95442b0e-e7b1-403f-9ad3-a6121c7b6b95" (UID: "95442b0e-e7b1-403f-9ad3-a6121c7b6b95"). InnerVolumeSpecName "kube-api-access-qghps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:44.613198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:44.613153    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-xks-ca" pod "95442b0e-e7b1-403f-9ad3-a6121c7b6b95" (UID: "95442b0e-e7b1-403f-9ad3-a6121c7b6b95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:07:44.712768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:44.712729    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qghps\" (UniqueName: \"kubernetes.io/projected/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-kube-api-access-qghps\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:44.712768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:44.712760    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95442b0e-e7b1-403f-9ad3-a6121c7b6b95-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:44.824040 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:07:44.823984    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:07:45.156545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:45.156512    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:45.326788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:45.326763    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"
Nov 02 13:07:45.326968 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:45.326793    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb" event={"ID":"95442b0e-e7b1-403f-9ad3-a6121c7b6b95","Type":"ContainerDied","Data":"99d82aba005ff49a95cde70ac78282a72597e0535ec96b57e520e366773c0728"}
Nov 02 13:07:45.326968 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:45.326838    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99d82aba005ff49a95cde70ac78282a72597e0535ec96b57e520e366773c0728"
Nov 02 13:07:46.965986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:46.965947    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:48.691148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:48.691116    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:50.381089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:50.381059    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:52.307423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:52.307381    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:53.546876 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:53.546842    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:53.670607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:53.670558    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:53.945828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:53.945788    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"]
Nov 02 13:07:53.947413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:53.947384    5879 cpu_manager.go:414] "containerMap: removing container" podUID="95442b0e-e7b1-403f-9ad3-a6121c7b6b95" containerName="revoke-keys"
Nov 02 13:07:53.947413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:53.947412    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95442b0e-e7b1-403f-9ad3-a6121c7b6b95" containerName="revoke-keys"
Nov 02 13:07:53.947641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:53.947578    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="95442b0e-e7b1-403f-9ad3-a6121c7b6b95" containerName="revoke-keys"
Nov 02 13:07:53.953579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:53.953551    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:53.986183 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:53.986147    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"]
Nov 02 13:07:54.053392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.053355    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.053574 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.053500    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8v4p\" (UniqueName: \"kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-kube-api-access-z8v4p\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.053574 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.053557    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.053776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.053760    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.116166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.116127    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:54.148474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.148444    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"]
Nov 02 13:07:54.153722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.153699    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:54.154860 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.154831    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.155669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.154929    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8v4p\" (UniqueName: \"kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-kube-api-access-z8v4p\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.155669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.155045    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.155669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.155241    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.156689 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.156659    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.156863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.156835    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.159294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.159265    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.173943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.173907    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"]
Nov 02 13:07:54.175151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.174969    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:07:54.181545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.181078    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8v4p\" (UniqueName: \"kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-kube-api-access-z8v4p\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.187259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.187229    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:07:54.257385 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.257351    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd8ql\" (UniqueName: \"kubernetes.io/projected/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-kube-api-access-nd8ql\") pod \"maas-api-revoke-sub-3e4bec90-sbgbw\" (UID: \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:54.257594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.257568    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e4bec90-sbgbw\" (UID: \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:54.257696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.257679    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e4bec90-sbgbw\" (UID: \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:54.308792 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.308752    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:54.359981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.359941    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8ql\" (UniqueName: \"kubernetes.io/projected/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-kube-api-access-nd8ql\") pod \"maas-api-revoke-sub-3e4bec90-sbgbw\" (UID: \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:54.360233 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.360115    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e4bec90-sbgbw\" (UID: \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:54.360233 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.360167    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e4bec90-sbgbw\" (UID: \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:54.360967 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.360941    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e4bec90-sbgbw\" (UID: \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:54.363214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.363190    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e4bec90-sbgbw\" (UID: \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:54.401251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.395681    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8ql\" (UniqueName: \"kubernetes.io/projected/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-kube-api-access-nd8ql\") pod \"maas-api-revoke-sub-3e4bec90-sbgbw\" (UID: \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:54.513083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.512974    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:54.628829 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:07:54.628276    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb130af_c8b8_4c29_8c88_61445e3077e0.slice/crio-fc1dc08e51674e738671caa58392e6da79b5f86ca6ced30fadcfd041d1375039 WatchSource:0}: Error finding container fc1dc08e51674e738671caa58392e6da79b5f86ca6ced30fadcfd041d1375039: Status 404 returned error can't find the container with id fc1dc08e51674e738671caa58392e6da79b5f86ca6ced30fadcfd041d1375039
Nov 02 13:07:54.629320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.629214    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"]
Nov 02 13:07:54.819744 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:07:54.819714    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a74ef2_5ad2_4e6d_964c_bf6094fc5ffb.slice/crio-6e71fd07d6673fa37634b7c739bf7e2a039219988a28ab50a6dc77d073357d2b WatchSource:0}: Error finding container 6e71fd07d6673fa37634b7c739bf7e2a039219988a28ab50a6dc77d073357d2b: Status 404 returned error can't find the container with id 6e71fd07d6673fa37634b7c739bf7e2a039219988a28ab50a6dc77d073357d2b
Nov 02 13:07:54.820624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:54.820552    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"]
Nov 02 13:07:55.417440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:55.417397    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw" event={"ID":"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb","Type":"ContainerStarted","Data":"d55e0adcd808b9319d5356d9ad82a514a66c72616b61e8ea3301f856d10e8d06"}
Nov 02 13:07:55.417644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:55.417479    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw" event={"ID":"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb","Type":"ContainerStarted","Data":"6e71fd07d6673fa37634b7c739bf7e2a039219988a28ab50a6dc77d073357d2b"}
Nov 02 13:07:55.419279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:55.419253    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd" event={"ID":"dfb130af-c8b8-4c29-8c88-61445e3077e0","Type":"ContainerStarted","Data":"d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288"}
Nov 02 13:07:55.419584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:55.419293    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd" event={"ID":"dfb130af-c8b8-4c29-8c88-61445e3077e0","Type":"ContainerStarted","Data":"fc1dc08e51674e738671caa58392e6da79b5f86ca6ced30fadcfd041d1375039"}
Nov 02 13:07:55.419584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:55.419390    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:07:55.462467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:55.462416    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw" podStartSLOduration=1.462402748 podStartE2EDuration="1.462402748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:07:55.461016426 +0000 UTC m=+2421.567719267" watchObservedRunningTime="2026-11-02 13:07:55.462402748 +0000 UTC m=+2421.569105599"
Nov 02 13:07:55.509286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:55.509217    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd" podStartSLOduration=2.509196987 podStartE2EDuration="2.509196987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:07:55.505812595 +0000 UTC m=+2421.612515448" watchObservedRunningTime="2026-11-02 13:07:55.509196987 +0000 UTC m=+2421.615899839"
Nov 02 13:07:55.920579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:55.920543    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:56.428435 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:56.428399    5879 generic.go:356] "Generic (PLEG): container finished" podID="23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb" containerID="d55e0adcd808b9319d5356d9ad82a514a66c72616b61e8ea3301f856d10e8d06" exitCode=0
Nov 02 13:07:56.428658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:56.428487    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw" event={"ID":"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb","Type":"ContainerDied","Data":"d55e0adcd808b9319d5356d9ad82a514a66c72616b61e8ea3301f856d10e8d06"}
Nov 02 13:07:57.597550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:57.597522    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:57.708285 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:57.708190    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-openshift-ca\") pod \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\" (UID: \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\") "
Nov 02 13:07:57.708475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:57.708298    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-kube-api-access-nd8ql\" (UniqueName: \"kubernetes.io/projected/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-kube-api-access-nd8ql\") pod \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\" (UID: \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\") "
Nov 02 13:07:57.708518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:57.708491    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-xks-ca\") pod \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\" (UID: \"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb\") "
Nov 02 13:07:57.708572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:57.708557    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-openshift-ca" pod "23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb" (UID: "23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:07:57.709056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:57.709033    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:57.710822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:57.710795    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-xks-ca" pod "23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb" (UID: "23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:07:57.710964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:57.710892    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-kube-api-access-nd8ql" pod "23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb" (UID: "23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb"). InnerVolumeSpecName "kube-api-access-nd8ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:07:57.810393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:57.810362    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:57.810393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:57.810390    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd8ql\" (UniqueName: \"kubernetes.io/projected/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb-kube-api-access-nd8ql\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:07:57.846906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:57.846864    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:07:58.446812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:58.446777    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"
Nov 02 13:07:58.447022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:58.446815    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw" event={"ID":"23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb","Type":"ContainerDied","Data":"6e71fd07d6673fa37634b7c739bf7e2a039219988a28ab50a6dc77d073357d2b"}
Nov 02 13:07:58.447022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:58.446866    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e71fd07d6673fa37634b7c739bf7e2a039219988a28ab50a6dc77d073357d2b"
Nov 02 13:07:59.605869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:07:59.605829    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:01.326078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:01.326041    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:03.113314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:03.113276    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:05.049091 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:05.049057    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:05.430458 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:05.430417    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"]
Nov 02 13:08:05.443555 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:05.443518    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"]
Nov 02 13:08:05.447584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:05.447555    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:08:05.547331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:05.547298    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:05.695895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:05.695812    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:05.816231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:05.816183    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:05.937071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:05.937032    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:06.810745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:06.810708    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:06.929903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:06.929875    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:07.073606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.073511    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:07.175057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.174983    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"]
Nov 02 13:08:07.175915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.175891    5879 cpu_manager.go:414] "containerMap: removing container" podUID="23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb" containerName="revoke-keys"
Nov 02 13:08:07.175915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.175913    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb" containerName="revoke-keys"
Nov 02 13:08:07.176116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.176105    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb" containerName="revoke-keys"
Nov 02 13:08:07.180276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.180249    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.199119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.199091    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"]
Nov 02 13:08:07.207027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.206963    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:07.208201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.208175    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"agc-cred-40b1f73dbc39\""
Nov 02 13:08:07.208422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.208379    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"payload-processing-external-model-e2e--ee70ab17-dockercfg-hk997\""
Nov 02 13:08:07.347087 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.346969    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.347247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.347153    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.347291 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.347244    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.347333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.347298    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w2fn\" (UniqueName: \"kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-kube-api-access-6w2fn\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.434178 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.434143    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:07.448830 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.448795    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.448983 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.448887    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.448983 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.448966    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.449124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.449057    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2fn\" (UniqueName: \"kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-kube-api-access-6w2fn\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.449215 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:07.449059    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-14fcb6/agc-cred-40b1f73dbc39: secret "agc-cred-40b1f73dbc39" not found
Nov 02 13:08:07.449286 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:07.449219    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw: secret "agc-cred-40b1f73dbc39" not found
Nov 02 13:08:07.449358 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:07.449339    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-provider-credentials podName:d4e9777d-a29f-46bc-a320-edc39ca941c4 nodeName:}" failed. No retries permitted until 2026-11-02 13:08:07.949309136 +0000 UTC m=+2434.056011988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw" (UID: "d4e9777d-a29f-46bc-a320-edc39ca941c4") : secret "agc-cred-40b1f73dbc39" not found
Nov 02 13:08:07.449688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.449667    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.449804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.449784    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.474769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.474731    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2fn\" (UniqueName: \"kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-kube-api-access-6w2fn\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.481851 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.481821    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"]
Nov 02 13:08:07.482319 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:07.482294    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw" podUID="d4e9777d-a29f-46bc-a320-edc39ca941c4"
Nov 02 13:08:07.527798 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.527760    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.538495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.538469    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.594757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.594726    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:07.651338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.651306    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-kube-api-access-6w2fn\" (UniqueName: \"kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-kube-api-access-6w2fn\") pod \"d4e9777d-a29f-46bc-a320-edc39ca941c4\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") "
Nov 02 13:08:07.651512 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.651383    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-plugins-config-volume\") pod \"d4e9777d-a29f-46bc-a320-edc39ca941c4\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") "
Nov 02 13:08:07.651512 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.651491    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-routing-overlay-volume\") pod \"d4e9777d-a29f-46bc-a320-edc39ca941c4\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") "
Nov 02 13:08:07.651816 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.651785    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-routing-overlay-volume" pod "d4e9777d-a29f-46bc-a320-edc39ca941c4" (UID: "d4e9777d-a29f-46bc-a320-edc39ca941c4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:07.651883 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.651823    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-plugins-config-volume" pod "d4e9777d-a29f-46bc-a320-edc39ca941c4" (UID: "d4e9777d-a29f-46bc-a320-edc39ca941c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:07.652070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.652057    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:07.652118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.652073    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4e9777d-a29f-46bc-a320-edc39ca941c4-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:07.653751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.653728    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-kube-api-access-6w2fn" pod "d4e9777d-a29f-46bc-a320-edc39ca941c4" (UID: "d4e9777d-a29f-46bc-a320-edc39ca941c4"). InnerVolumeSpecName "kube-api-access-6w2fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:07.753573 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.753539    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w2fn\" (UniqueName: \"kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-kube-api-access-6w2fn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:07.958172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:07.958074    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw\" (UID: \"d4e9777d-a29f-46bc-a320-edc39ca941c4\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:07.958639 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:07.958251    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-14fcb6/agc-cred-40b1f73dbc39: secret "agc-cred-40b1f73dbc39" not found
Nov 02 13:08:07.958639 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:07.958272    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw: secret "agc-cred-40b1f73dbc39" not found
Nov 02 13:08:07.958639 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:07.958339    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-provider-credentials podName:d4e9777d-a29f-46bc-a320-edc39ca941c4 nodeName:}" failed. No retries permitted until 2026-11-02 13:08:08.958322906 +0000 UTC m=+2435.065025735 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw" (UID: "d4e9777d-a29f-46bc-a320-edc39ca941c4") : secret "agc-cred-40b1f73dbc39" not found
Nov 02 13:08:08.535573 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:08.535542    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"
Nov 02 13:08:08.632048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:08.627720    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"]
Nov 02 13:08:08.634260 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:08.634227    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7cf7gb6mw"]
Nov 02 13:08:08.770222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:08.770180    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4e9777d-a29f-46bc-a320-edc39ca941c4-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:09.061895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:09.061859    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:10.018639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.018589    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e9777d-a29f-46bc-a320-edc39ca941c4" path="/var/lib/kubelet/pods/d4e9777d-a29f-46bc-a320-edc39ca941c4/volumes"
Nov 02 13:08:10.430836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.430784    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v" podUID="d3d3caee-b5f3-4755-ab24-fef15f7f2fcd" containerName="payload-processing" containerID="cri-o://8ea45952d8f28ade9ddd0a88579d5347ef201ea61ccdf9114afa3d0a6c39c59f" gracePeriod=25
Nov 02 13:08:10.444631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.444586    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd" podUID="dfb130af-c8b8-4c29-8c88-61445e3077e0" containerName="payload-processing" containerID="cri-o://d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288" gracePeriod=25
Nov 02 13:08:10.552205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.552169    5879 generic.go:356] "Generic (PLEG): container finished" podID="d3d3caee-b5f3-4755-ab24-fef15f7f2fcd" containerID="8ea45952d8f28ade9ddd0a88579d5347ef201ea61ccdf9114afa3d0a6c39c59f" exitCode=0
Nov 02 13:08:10.552336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.552238    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v" event={"ID":"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd","Type":"ContainerDied","Data":"8ea45952d8f28ade9ddd0a88579d5347ef201ea61ccdf9114afa3d0a6c39c59f"}
Nov 02 13:08:10.736376 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.736346    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:08:10.755255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.755222    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:10.766779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.766750    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:08:10.799027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.798964    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-kube-api-access-rgsv4\" (UniqueName: \"kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-kube-api-access-rgsv4\") pod \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") "
Nov 02 13:08:10.799275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.799134    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-plugins-config-volume\") pod \"dfb130af-c8b8-4c29-8c88-61445e3077e0\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") "
Nov 02 13:08:10.799275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.799185    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-provider-credentials\") pod \"dfb130af-c8b8-4c29-8c88-61445e3077e0\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") "
Nov 02 13:08:10.799275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.799253    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-routing-overlay-volume\") pod \"dfb130af-c8b8-4c29-8c88-61445e3077e0\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") "
Nov 02 13:08:10.799507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.799289    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-kube-api-access-z8v4p\" (UniqueName: \"kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-kube-api-access-z8v4p\") pod \"dfb130af-c8b8-4c29-8c88-61445e3077e0\" (UID: \"dfb130af-c8b8-4c29-8c88-61445e3077e0\") "
Nov 02 13:08:10.799507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.799310    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-provider-credentials\") pod \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") "
Nov 02 13:08:10.799507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.799337    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-plugins-config-volume\") pod \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") "
Nov 02 13:08:10.799769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.799724    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-routing-overlay-volume" pod "dfb130af-c8b8-4c29-8c88-61445e3077e0" (UID: "dfb130af-c8b8-4c29-8c88-61445e3077e0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:10.800166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.800121    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-plugins-config-volume" pod "d3d3caee-b5f3-4755-ab24-fef15f7f2fcd" (UID: "d3d3caee-b5f3-4755-ab24-fef15f7f2fcd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:10.800292 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.800179    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:10.803730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.800585    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-plugins-config-volume" pod "dfb130af-c8b8-4c29-8c88-61445e3077e0" (UID: "dfb130af-c8b8-4c29-8c88-61445e3077e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:10.803730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.802569    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-kube-api-access-rgsv4" pod "d3d3caee-b5f3-4755-ab24-fef15f7f2fcd" (UID: "d3d3caee-b5f3-4755-ab24-fef15f7f2fcd"). InnerVolumeSpecName "kube-api-access-rgsv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:10.803730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.803124    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-provider-credentials" pod "d3d3caee-b5f3-4755-ab24-fef15f7f2fcd" (UID: "d3d3caee-b5f3-4755-ab24-fef15f7f2fcd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:10.803730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.803221    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-provider-credentials" pod "dfb130af-c8b8-4c29-8c88-61445e3077e0" (UID: "dfb130af-c8b8-4c29-8c88-61445e3077e0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:10.804577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.804068    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-kube-api-access-z8v4p" pod "dfb130af-c8b8-4c29-8c88-61445e3077e0" (UID: "dfb130af-c8b8-4c29-8c88-61445e3077e0"). InnerVolumeSpecName "kube-api-access-z8v4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:10.900586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.900552    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-routing-overlay-volume\") pod \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\" (UID: \"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd\") "
Nov 02 13:08:10.900925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.900901    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-routing-overlay-volume" pod "d3d3caee-b5f3-4755-ab24-fef15f7f2fcd" (UID: "d3d3caee-b5f3-4755-ab24-fef15f7f2fcd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:10.901083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.901068    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgsv4\" (UniqueName: \"kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-kube-api-access-rgsv4\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:10.901139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.901085    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb130af-c8b8-4c29-8c88-61445e3077e0-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:10.901139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.901094    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:10.901139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.901103    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8v4p\" (UniqueName: \"kubernetes.io/projected/dfb130af-c8b8-4c29-8c88-61445e3077e0-kube-api-access-z8v4p\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:10.901139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.901111    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:10.901139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.901119    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:10.901139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:10.901127    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:11.561748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.561714    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"
Nov 02 13:08:11.562263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.561753    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v" event={"ID":"d3d3caee-b5f3-4755-ab24-fef15f7f2fcd","Type":"ContainerDied","Data":"2520f59cb8e20ac7e06753c4433f639c9fa53f565029b60cf30b4182cab714e1"}
Nov 02 13:08:11.562263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.561829    5879 scope.go:122] "RemoveContainer" containerID="8ea45952d8f28ade9ddd0a88579d5347ef201ea61ccdf9114afa3d0a6c39c59f"
Nov 02 13:08:11.563336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.563316    5879 generic.go:356] "Generic (PLEG): container finished" podID="dfb130af-c8b8-4c29-8c88-61445e3077e0" containerID="d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288" exitCode=0
Nov 02 13:08:11.563430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.563355    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"
Nov 02 13:08:11.563430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.563397    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd" event={"ID":"dfb130af-c8b8-4c29-8c88-61445e3077e0","Type":"ContainerDied","Data":"d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288"}
Nov 02 13:08:11.563505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.563453    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd" event={"ID":"dfb130af-c8b8-4c29-8c88-61445e3077e0","Type":"ContainerDied","Data":"fc1dc08e51674e738671caa58392e6da79b5f86ca6ced30fadcfd041d1375039"}
Nov 02 13:08:11.575135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.575107    5879 scope.go:122] "RemoveContainer" containerID="d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288"
Nov 02 13:08:11.585373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.585347    5879 scope.go:122] "RemoveContainer" containerID="d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288"
Nov 02 13:08:11.585656 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:11.585637    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288\": container with ID starting with d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288 not found: ID does not exist" containerID="d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288"
Nov 02 13:08:11.585708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.585666    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288"} err="failed to get container status \"d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288\": rpc error: code = NotFound desc = could not find container \"d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288\": container with ID starting with d226a5a5bbe32fa2192e9fa2a605ba06ee13e02941ed8051893fdc926462a288 not found: ID does not exist"
Nov 02 13:08:11.623178 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.623145    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"]
Nov 02 13:08:11.636655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.636622    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-6d9fpw89v"]
Nov 02 13:08:11.670695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.670662    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"]
Nov 02 13:08:11.678695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:11.678662    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7ddcc8zrd"]
Nov 02 13:08:12.018103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:12.018070    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d3caee-b5f3-4755-ab24-fef15f7f2fcd" path="/var/lib/kubelet/pods/d3d3caee-b5f3-4755-ab24-fef15f7f2fcd/volumes"
Nov 02 13:08:12.018557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:12.018536    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb130af-c8b8-4c29-8c88-61445e3077e0" path="/var/lib/kubelet/pods/dfb130af-c8b8-4c29-8c88-61445e3077e0/volumes"
Nov 02 13:08:12.531119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:12.531081    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:13.634706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:13.634622    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:13.765281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:13.765247    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:13.887062 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:13.886941    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:14.307204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:14.307122    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:14.822316 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:14.822279    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:08:16.058059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:16.057988    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:17.975769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:17.975735    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:19.780837 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:19.780800    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:20.205003 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:20.204971    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:20.362132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:20.362093    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:21.214563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:21.214515    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:21.351732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:21.351698    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:21.842308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:21.842274    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:22.716677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:22.716635    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:23.291037 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.290974    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"]
Nov 02 13:08:23.291841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.291825    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d3d3caee-b5f3-4755-ab24-fef15f7f2fcd" containerName="payload-processing"
Nov 02 13:08:23.291902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.291844    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d3caee-b5f3-4755-ab24-fef15f7f2fcd" containerName="payload-processing"
Nov 02 13:08:23.291902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.291858    5879 cpu_manager.go:414] "containerMap: removing container" podUID="dfb130af-c8b8-4c29-8c88-61445e3077e0" containerName="payload-processing"
Nov 02 13:08:23.291902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.291862    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb130af-c8b8-4c29-8c88-61445e3077e0" containerName="payload-processing"
Nov 02 13:08:23.292043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.292028    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d3caee-b5f3-4755-ab24-fef15f7f2fcd" containerName="payload-processing"
Nov 02 13:08:23.292093 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.292050    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb130af-c8b8-4c29-8c88-61445e3077e0" containerName="payload-processing"
Nov 02 13:08:23.297856 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.297830    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.319703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.319677    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 02 13:08:23.320287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.320268    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 02 13:08:23.333536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.333509    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c12c80598492\""
Nov 02 13:08:23.333719 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.333602    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-vjcqq\""
Nov 02 13:08:23.342266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.342239    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"]
Nov 02 13:08:23.386297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.386261    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-provider-credentials\") pod \"payload-processing-external-model-6f96cb5db5-7kr55\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") " pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.386513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.386478    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2967\" (UniqueName: \"kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-kube-api-access-v2967\") pod \"payload-processing-external-model-6f96cb5db5-7kr55\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") " pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.386620 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.386599    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-routing-overlay-volume\") pod \"payload-processing-external-model-6f96cb5db5-7kr55\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") " pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.386697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.386658    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-plugins-config-volume\") pod \"payload-processing-external-model-6f96cb5db5-7kr55\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") " pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.417915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.417877    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 02 13:08:23.418115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.417937    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 02 13:08:23.487493 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.487456    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2967\" (UniqueName: \"kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-kube-api-access-v2967\") pod \"payload-processing-external-model-6f96cb5db5-7kr55\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") " pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.487688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.487548    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-routing-overlay-volume\") pod \"payload-processing-external-model-6f96cb5db5-7kr55\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") " pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.487688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.487592    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-plugins-config-volume\") pod \"payload-processing-external-model-6f96cb5db5-7kr55\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") " pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.487832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.487698    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-provider-credentials\") pod \"payload-processing-external-model-6f96cb5db5-7kr55\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") " pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.488421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.488398    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-plugins-config-volume\") pod \"payload-processing-external-model-6f96cb5db5-7kr55\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") " pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.488523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.488443    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-routing-overlay-volume\") pod \"payload-processing-external-model-6f96cb5db5-7kr55\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") " pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.490260 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.490239    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-provider-credentials\") pod \"payload-processing-external-model-6f96cb5db5-7kr55\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") " pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.515277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.515246    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2967\" (UniqueName: \"kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-kube-api-access-v2967\") pod \"payload-processing-external-model-6f96cb5db5-7kr55\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") " pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.614784 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.614760    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:23.767938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.767904    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:23.965539 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:08:23.965506    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4217d62_9037_41fd_9279_5c59120431a3.slice/crio-22db9384543d77defb4783dedcdd36d745ac77ecfd047f77cff71d87e1cef19b WatchSource:0}: Error finding container 22db9384543d77defb4783dedcdd36d745ac77ecfd047f77cff71d87e1cef19b: Status 404 returned error can't find the container with id 22db9384543d77defb4783dedcdd36d745ac77ecfd047f77cff71d87e1cef19b
Nov 02 13:08:23.966046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:23.965976    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"]
Nov 02 13:08:24.681616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:24.681576    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55" event={"ID":"f4217d62-9037-41fd-9279-5c59120431a3","Type":"ContainerStarted","Data":"8d741c4a9e9d9969505687746a87cfc174bfbf5a77077ff52046e544f7dbf9ce"}
Nov 02 13:08:24.681808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:24.681636    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55" event={"ID":"f4217d62-9037-41fd-9279-5c59120431a3","Type":"ContainerStarted","Data":"22db9384543d77defb4783dedcdd36d745ac77ecfd047f77cff71d87e1cef19b"}
Nov 02 13:08:24.681808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:24.681666    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:24.731402 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:24.731338    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55" podStartSLOduration=1.731323976 podStartE2EDuration="1.731323976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:08:24.72803665 +0000 UTC m=+2450.834739502" watchObservedRunningTime="2026-11-02 13:08:24.731323976 +0000 UTC m=+2450.838026826"
Nov 02 13:08:25.838360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:25.838326    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:27.612631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:27.612589    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:29.477856 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:29.477816    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:31.183228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:31.183197    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:32.974727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:32.974697    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:33.987214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:33.987183    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:34.371573 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.371538    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"]
Nov 02 13:08:34.376572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.376550    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:34.396131 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.396102    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"]
Nov 02 13:08:34.406566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.405444    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:08:34.406566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.405463    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:08:34.464191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.464156    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c20986d2-4qwn6\" (UID: \"f2e7ad32-cade-4621-846f-d1a057f39c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:34.464446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.464428    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c20986d2-4qwn6\" (UID: \"f2e7ad32-cade-4621-846f-d1a057f39c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:34.464631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.464611    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d579\" (UniqueName: \"kubernetes.io/projected/f2e7ad32-cade-4621-846f-d1a057f39c47-kube-api-access-7d579\") pod \"maas-api-revoke-sub-c20986d2-4qwn6\" (UID: \"f2e7ad32-cade-4621-846f-d1a057f39c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:34.566183 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.566150    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c20986d2-4qwn6\" (UID: \"f2e7ad32-cade-4621-846f-d1a057f39c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:34.566356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.566271    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d579\" (UniqueName: \"kubernetes.io/projected/f2e7ad32-cade-4621-846f-d1a057f39c47-kube-api-access-7d579\") pod \"maas-api-revoke-sub-c20986d2-4qwn6\" (UID: \"f2e7ad32-cade-4621-846f-d1a057f39c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:34.566356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.566300    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c20986d2-4qwn6\" (UID: \"f2e7ad32-cade-4621-846f-d1a057f39c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:34.567063 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.567039    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c20986d2-4qwn6\" (UID: \"f2e7ad32-cade-4621-846f-d1a057f39c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:34.568710 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.568691    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c20986d2-4qwn6\" (UID: \"f2e7ad32-cade-4621-846f-d1a057f39c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:34.592275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.592249    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d579\" (UniqueName: \"kubernetes.io/projected/f2e7ad32-cade-4621-846f-d1a057f39c47-kube-api-access-7d579\") pod \"maas-api-revoke-sub-c20986d2-4qwn6\" (UID: \"f2e7ad32-cade-4621-846f-d1a057f39c47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:34.694444 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.694365    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:34.770030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:34.769873    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:35.005822 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:08:35.005791    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e7ad32_cade_4621_846f_d1a057f39c47.slice/crio-2d0f4bd1fee46da89fd86cf942c01f0f24145bd38c5cf22b3f2fe245c3bfafdd WatchSource:0}: Error finding container 2d0f4bd1fee46da89fd86cf942c01f0f24145bd38c5cf22b3f2fe245c3bfafdd: Status 404 returned error can't find the container with id 2d0f4bd1fee46da89fd86cf942c01f0f24145bd38c5cf22b3f2fe245c3bfafdd
Nov 02 13:08:35.007275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:35.007220    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"]
Nov 02 13:08:35.696683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:35.696653    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:08:35.777325 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:35.777273    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6" event={"ID":"f2e7ad32-cade-4621-846f-d1a057f39c47","Type":"ContainerStarted","Data":"48bbf0bcd68e1e1f150de4fff1d0e7aef6182a43d6aa72afc1af38bb31086d43"}
Nov 02 13:08:35.777542 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:35.777355    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6" event={"ID":"f2e7ad32-cade-4621-846f-d1a057f39c47","Type":"ContainerStarted","Data":"2d0f4bd1fee46da89fd86cf942c01f0f24145bd38c5cf22b3f2fe245c3bfafdd"}
Nov 02 13:08:35.824137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:35.824083    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6" podStartSLOduration=1.824069143 podStartE2EDuration="1.824069143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:08:35.820766902 +0000 UTC m=+2461.927469754" watchObservedRunningTime="2026-11-02 13:08:35.824069143 +0000 UTC m=+2461.930771993"
Nov 02 13:08:35.944687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:35.944644    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"]
Nov 02 13:08:35.944934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:35.944892    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerName="istio-proxy" containerID="cri-o://8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c" gracePeriod=30
Nov 02 13:08:36.576921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:36.576886    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:36.787900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:36.787809    5879 generic.go:356] "Generic (PLEG): container finished" podID="f2e7ad32-cade-4621-846f-d1a057f39c47" containerID="48bbf0bcd68e1e1f150de4fff1d0e7aef6182a43d6aa72afc1af38bb31086d43" exitCode=0
Nov 02 13:08:36.788075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:36.787945    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6" event={"ID":"f2e7ad32-cade-4621-846f-d1a057f39c47","Type":"ContainerDied","Data":"48bbf0bcd68e1e1f150de4fff1d0e7aef6182a43d6aa72afc1af38bb31086d43"}
Nov 02 13:08:37.962666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:37.962640    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:38.118252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.118223    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e7ad32-cade-4621-846f-d1a057f39c47-kube-api-access-7d579\" (UniqueName: \"kubernetes.io/projected/f2e7ad32-cade-4621-846f-d1a057f39c47-kube-api-access-7d579\") pod \"f2e7ad32-cade-4621-846f-d1a057f39c47\" (UID: \"f2e7ad32-cade-4621-846f-d1a057f39c47\") "
Nov 02 13:08:38.118433 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.118270    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-xks-ca\") pod \"f2e7ad32-cade-4621-846f-d1a057f39c47\" (UID: \"f2e7ad32-cade-4621-846f-d1a057f39c47\") "
Nov 02 13:08:38.118496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.118479    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-openshift-ca\") pod \"f2e7ad32-cade-4621-846f-d1a057f39c47\" (UID: \"f2e7ad32-cade-4621-846f-d1a057f39c47\") "
Nov 02 13:08:38.118876 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.118839    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-openshift-ca" pod "f2e7ad32-cade-4621-846f-d1a057f39c47" (UID: "f2e7ad32-cade-4621-846f-d1a057f39c47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:38.119452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.119423    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:38.120637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.120612    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e7ad32-cade-4621-846f-d1a057f39c47-kube-api-access-7d579" pod "f2e7ad32-cade-4621-846f-d1a057f39c47" (UID: "f2e7ad32-cade-4621-846f-d1a057f39c47"). InnerVolumeSpecName "kube-api-access-7d579". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:38.120711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.120680    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-xks-ca" pod "f2e7ad32-cade-4621-846f-d1a057f39c47" (UID: "f2e7ad32-cade-4621-846f-d1a057f39c47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:08:38.220674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.220639    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d579\" (UniqueName: \"kubernetes.io/projected/f2e7ad32-cade-4621-846f-d1a057f39c47-kube-api-access-7d579\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:38.220674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.220670    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e7ad32-cade-4621-846f-d1a057f39c47-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:38.357572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.357535    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:38.574323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.574232    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:38.805576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.805549    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"
Nov 02 13:08:38.805733 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.805591    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6" event={"ID":"f2e7ad32-cade-4621-846f-d1a057f39c47","Type":"ContainerDied","Data":"2d0f4bd1fee46da89fd86cf942c01f0f24145bd38c5cf22b3f2fe245c3bfafdd"}
Nov 02 13:08:38.805733 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:38.805646    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d0f4bd1fee46da89fd86cf942c01f0f24145bd38c5cf22b3f2fe245c3bfafdd"
Nov 02 13:08:38.873817 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:38.873786    5879 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e7ad32_cade_4621_846f_d1a057f39c47.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 13:08:39.253683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.253603    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:39.391166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.391123    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:39.657118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.657081    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"]
Nov 02 13:08:39.657962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.657930    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f2e7ad32-cade-4621-846f-d1a057f39c47" containerName="revoke-keys"
Nov 02 13:08:39.657962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.657957    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e7ad32-cade-4621-846f-d1a057f39c47" containerName="revoke-keys"
Nov 02 13:08:39.658202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.658163    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e7ad32-cade-4621-846f-d1a057f39c47" containerName="revoke-keys"
Nov 02 13:08:39.662125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.662102    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.682618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.682588    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"]
Nov 02 13:08:39.688528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.688419    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:08:39.688743 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.688716    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:08:39.847014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.846947    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rczdm\" (UniqueName: \"kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-kube-api-access-rczdm\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.847198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.847129    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.847198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.847177    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.847297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.847279    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.948586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.948502    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczdm\" (UniqueName: \"kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-kube-api-access-rczdm\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.948586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.948564    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.948806 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.948605    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.948806 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.948664    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.949388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.949333    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.949388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.949373    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.951559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.951538    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.973967 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.973933    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczdm\" (UniqueName: \"kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-kube-api-access-rczdm\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:39.979782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:39.979749    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:40.295625 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:08:40.295590    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ba9b93_367e_4b2a_83c9_29cddfdda3ac.slice/crio-f602c80d3420ac7ac78d190faf60684406d81572d435c2874764d1c37671f7b0 WatchSource:0}: Error finding container f602c80d3420ac7ac78d190faf60684406d81572d435c2874764d1c37671f7b0: Status 404 returned error can't find the container with id f602c80d3420ac7ac78d190faf60684406d81572d435c2874764d1c37671f7b0
Nov 02 13:08:40.298021 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:40.297977    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"]
Nov 02 13:08:40.362021 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:40.361981    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:40.828863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:40.828826    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h" event={"ID":"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac","Type":"ContainerStarted","Data":"07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063"}
Nov 02 13:08:40.829090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:40.828890    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h" event={"ID":"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac","Type":"ContainerStarted","Data":"f602c80d3420ac7ac78d190faf60684406d81572d435c2874764d1c37671f7b0"}
Nov 02 13:08:40.829090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:40.829036    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:40.875053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:40.874975    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h" podStartSLOduration=1.8749615689999999 podStartE2EDuration="1.874961569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:08:40.872064143 +0000 UTC m=+2466.978766993" watchObservedRunningTime="2026-11-02 13:08:40.874961569 +0000 UTC m=+2466.981664419"
Nov 02 13:08:40.956276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:40.956240    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:41.252130 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.252101    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 13:08:41.369628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.369599    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-podinfo\") pod \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") "
Nov 02 13:08:41.370150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.369678    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-data\") pod \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") "
Nov 02 13:08:41.370150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.369791    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-credential-socket\") pod \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") "
Nov 02 13:08:41.370150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.369840    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-socket\") pod \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") "
Nov 02 13:08:41.370150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.369865    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-kube-api-access-t6vq9\" (UniqueName: \"kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-kube-api-access-t6vq9\") pod \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") "
Nov 02 13:08:41.370150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.369920    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-certs\") pod \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") "
Nov 02 13:08:41.370150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.370008    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istiod-ca-cert\") pod \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") "
Nov 02 13:08:41.370150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.370060    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-envoy\") pod \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") "
Nov 02 13:08:41.370150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.370052    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-data" pod "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" (UID: "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:41.370150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.370134    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-token\" (UniqueName: \"kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-token\") pod \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\" (UID: \"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8\") "
Nov 02 13:08:41.370625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.370137    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-socket" pod "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" (UID: "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:41.370625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.370152    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-credential-socket" pod "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" (UID: "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:41.370625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.370237    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-certs" pod "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" (UID: "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:41.370625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.370375    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istiod-ca-cert" pod "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" (UID: "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:41.370957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.370850    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:41.371040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.371016    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:41.371099 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.371047    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:41.371099 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.371062    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:41.371099 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.371075    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:41.372455 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.372418    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-envoy" pod "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" (UID: "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:08:41.372591 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.372567    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-token" pod "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" (UID: "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:41.372755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.372734    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-kube-api-access-t6vq9" pod "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" (UID: "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8"). InnerVolumeSpecName "kube-api-access-t6vq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:41.373116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.373093    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-podinfo" pod "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" (UID: "4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:08:41.472400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.472364    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:41.472400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.472402    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:41.472625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.472412    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6vq9\" (UniqueName: \"kubernetes.io/projected/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-kube-api-access-t6vq9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:41.472625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.472421    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:41.839075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.838961    5879 generic.go:356] "Generic (PLEG): container finished" podID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerID="8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c" exitCode=0
Nov 02 13:08:41.839075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.839035    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"
Nov 02 13:08:41.839320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.839127    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" event={"ID":"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8","Type":"ContainerDied","Data":"8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c"}
Nov 02 13:08:41.839320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.839175    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg" event={"ID":"4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8","Type":"ContainerDied","Data":"9746f385e3aebfc6163158ee871fe64912e74b3ea41b974beb94c599b387b5a6"}
Nov 02 13:08:41.839320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.839205    5879 scope.go:122] "RemoveContainer" containerID="8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c"
Nov 02 13:08:41.853350 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.853328    5879 scope.go:122] "RemoveContainer" containerID="8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c"
Nov 02 13:08:41.853725 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:41.853706    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c\": container with ID starting with 8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c not found: ID does not exist" containerID="8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c"
Nov 02 13:08:41.853774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.853734    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c"} err="failed to get container status \"8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c\": rpc error: code = NotFound desc = could not find container \"8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c\": container with ID starting with 8202bd8ce2416ea8e7aef1c04416ca81f2f2a72cd74f41f10815116d85fabb7c not found: ID does not exist"
Nov 02 13:08:41.898412 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.898378    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"]
Nov 02 13:08:41.911750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:41.911720    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-qqhmg"]
Nov 02 13:08:42.018890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:42.018855    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" path="/var/lib/kubelet/pods/4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8/volumes"
Nov 02 13:08:42.261951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:42.261917    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:44.164226 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:44.164184    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:44.824412 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:44.824374    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:08:45.810609 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:45.810561    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"]
Nov 02 13:08:45.846531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:45.839844    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:46.040118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:46.040083    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:46.427570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:46.427539    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:47.472142 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.472104    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:47.610410 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.610374    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:47.612484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.612461    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"]
Nov 02 13:08:47.613658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.613636    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerName="istio-proxy"
Nov 02 13:08:47.613658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.613660    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerName="istio-proxy"
Nov 02 13:08:47.613930 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.613916    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3b2efe-fbb1-4c96-abf6-d9bd3b31ebb8" containerName="istio-proxy"
Nov 02 13:08:47.619208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.619180    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:47.639051 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.639020    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"]
Nov 02 13:08:47.647966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.647914    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:08:47.648202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.647980    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:08:47.658886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.658852    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd569777-fmgqq\" (UID: \"e0e7e43d-334c-4367-8642-5669d5df4c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:47.659109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.658921    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd569777-fmgqq\" (UID: \"e0e7e43d-334c-4367-8642-5669d5df4c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:47.659167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.659141    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7c4l\" (UniqueName: \"kubernetes.io/projected/e0e7e43d-334c-4367-8642-5669d5df4c8e-kube-api-access-p7c4l\") pod \"maas-api-revoke-sub-cd569777-fmgqq\" (UID: \"e0e7e43d-334c-4367-8642-5669d5df4c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:47.760238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.760147    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7c4l\" (UniqueName: \"kubernetes.io/projected/e0e7e43d-334c-4367-8642-5669d5df4c8e-kube-api-access-p7c4l\") pod \"maas-api-revoke-sub-cd569777-fmgqq\" (UID: \"e0e7e43d-334c-4367-8642-5669d5df4c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:47.760429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.760156    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:47.760429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.760320    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd569777-fmgqq\" (UID: \"e0e7e43d-334c-4367-8642-5669d5df4c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:47.760429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.760358    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd569777-fmgqq\" (UID: \"e0e7e43d-334c-4367-8642-5669d5df4c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:47.760986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.760961    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd569777-fmgqq\" (UID: \"e0e7e43d-334c-4367-8642-5669d5df4c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:47.762961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.762941    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd569777-fmgqq\" (UID: \"e0e7e43d-334c-4367-8642-5669d5df4c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:47.785266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.785227    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7c4l\" (UniqueName: \"kubernetes.io/projected/e0e7e43d-334c-4367-8642-5669d5df4c8e-kube-api-access-p7c4l\") pod \"maas-api-revoke-sub-cd569777-fmgqq\" (UID: \"e0e7e43d-334c-4367-8642-5669d5df4c8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:47.939193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:47.939153    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:48.254037 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:08:48.253961    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e7e43d_334c_4367_8642_5669d5df4c8e.slice/crio-2487d6ce6e5ec41746867cddb28a5148a1879bd0e4ee46703049fef804dba1d9 WatchSource:0}: Error finding container 2487d6ce6e5ec41746867cddb28a5148a1879bd0e4ee46703049fef804dba1d9: Status 404 returned error can't find the container with id 2487d6ce6e5ec41746867cddb28a5148a1879bd0e4ee46703049fef804dba1d9
Nov 02 13:08:48.254960 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:48.254936    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"]
Nov 02 13:08:48.898165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:48.898121    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq" event={"ID":"e0e7e43d-334c-4367-8642-5669d5df4c8e","Type":"ContainerStarted","Data":"221d91b913edfa572046c312f57c2745aaf9b11c379a90e0a92d36f9f8f74cb1"}
Nov 02 13:08:48.898562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:48.898198    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq" event={"ID":"e0e7e43d-334c-4367-8642-5669d5df4c8e","Type":"ContainerStarted","Data":"2487d6ce6e5ec41746867cddb28a5148a1879bd0e4ee46703049fef804dba1d9"}
Nov 02 13:08:48.944367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:48.944254    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq" podStartSLOduration=1.944235282 podStartE2EDuration="1.944235282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:08:48.943338435 +0000 UTC m=+2475.050041296" watchObservedRunningTime="2026-11-02 13:08:48.944235282 +0000 UTC m=+2475.050938133"
Nov 02 13:08:49.406154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:49.406123    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:49.907775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:49.907739    5879 generic.go:356] "Generic (PLEG): container finished" podID="e0e7e43d-334c-4367-8642-5669d5df4c8e" containerID="221d91b913edfa572046c312f57c2745aaf9b11c379a90e0a92d36f9f8f74cb1" exitCode=0
Nov 02 13:08:49.908200 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:49.907821    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq" event={"ID":"e0e7e43d-334c-4367-8642-5669d5df4c8e","Type":"ContainerDied","Data":"221d91b913edfa572046c312f57c2745aaf9b11c379a90e0a92d36f9f8f74cb1"}
Nov 02 13:08:50.811433 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:50.811382    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw" podUID="556a66bf-9cc7-49ec-b9b5-066c7976e718" containerName="payload-processing" containerID="cri-o://c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543" gracePeriod=25
Nov 02 13:08:51.109483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.109454    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:51.118207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.118180    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:08:51.188049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.188018    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:51.209107 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.209062    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-openshift-ca\") pod \"e0e7e43d-334c-4367-8642-5669d5df4c8e\" (UID: \"e0e7e43d-334c-4367-8642-5669d5df4c8e\") "
Nov 02 13:08:51.209977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.209841    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-openshift-ca" pod "e0e7e43d-334c-4367-8642-5669d5df4c8e" (UID: "e0e7e43d-334c-4367-8642-5669d5df4c8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:51.210348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.210329    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-routing-overlay-volume\") pod \"556a66bf-9cc7-49ec-b9b5-066c7976e718\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") "
Nov 02 13:08:51.211140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.210979    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-kube-api-access-77nhd\" (UniqueName: \"kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-kube-api-access-77nhd\") pod \"556a66bf-9cc7-49ec-b9b5-066c7976e718\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") "
Nov 02 13:08:51.211356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.210884    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-routing-overlay-volume" pod "556a66bf-9cc7-49ec-b9b5-066c7976e718" (UID: "556a66bf-9cc7-49ec-b9b5-066c7976e718"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:51.211612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.211596    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e7e43d-334c-4367-8642-5669d5df4c8e-kube-api-access-p7c4l\" (UniqueName: \"kubernetes.io/projected/e0e7e43d-334c-4367-8642-5669d5df4c8e-kube-api-access-p7c4l\") pod \"e0e7e43d-334c-4367-8642-5669d5df4c8e\" (UID: \"e0e7e43d-334c-4367-8642-5669d5df4c8e\") "
Nov 02 13:08:51.212144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.212123    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-xks-ca\") pod \"e0e7e43d-334c-4367-8642-5669d5df4c8e\" (UID: \"e0e7e43d-334c-4367-8642-5669d5df4c8e\") "
Nov 02 13:08:51.212683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.212663    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-provider-credentials\" (UniqueName: \"kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-provider-credentials\") pod \"556a66bf-9cc7-49ec-b9b5-066c7976e718\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") "
Nov 02 13:08:51.213253 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.213229    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-plugins-config-volume\") pod \"556a66bf-9cc7-49ec-b9b5-066c7976e718\" (UID: \"556a66bf-9cc7-49ec-b9b5-066c7976e718\") "
Nov 02 13:08:51.214034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.213976    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:51.216135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.215621    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:51.216135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.214493    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-plugins-config-volume" pod "556a66bf-9cc7-49ec-b9b5-066c7976e718" (UID: "556a66bf-9cc7-49ec-b9b5-066c7976e718"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:51.219516 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.219473    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-xks-ca" pod "e0e7e43d-334c-4367-8642-5669d5df4c8e" (UID: "e0e7e43d-334c-4367-8642-5669d5df4c8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:08:51.220976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.220934    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e7e43d-334c-4367-8642-5669d5df4c8e-kube-api-access-p7c4l" pod "e0e7e43d-334c-4367-8642-5669d5df4c8e" (UID: "e0e7e43d-334c-4367-8642-5669d5df4c8e"). InnerVolumeSpecName "kube-api-access-p7c4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:51.221235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.221137    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-provider-credentials" pod "556a66bf-9cc7-49ec-b9b5-066c7976e718" (UID: "556a66bf-9cc7-49ec-b9b5-066c7976e718"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:51.221235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.221175    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-kube-api-access-77nhd" pod "556a66bf-9cc7-49ec-b9b5-066c7976e718" (UID: "556a66bf-9cc7-49ec-b9b5-066c7976e718"). InnerVolumeSpecName "kube-api-access-77nhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:51.317198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.317164    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/556a66bf-9cc7-49ec-b9b5-066c7976e718-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:51.317198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.317195    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77nhd\" (UniqueName: \"kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-kube-api-access-77nhd\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:51.317198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.317204    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7c4l\" (UniqueName: \"kubernetes.io/projected/e0e7e43d-334c-4367-8642-5669d5df4c8e-kube-api-access-p7c4l\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:51.317461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.317214    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0e7e43d-334c-4367-8642-5669d5df4c8e-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:51.317461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.317222    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/556a66bf-9cc7-49ec-b9b5-066c7976e718-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:51.843809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.843778    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:08:51.927635 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.927600    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"]
Nov 02 13:08:51.928124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.928099    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"
Nov 02 13:08:51.928273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.928180    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq" event={"ID":"e0e7e43d-334c-4367-8642-5669d5df4c8e","Type":"ContainerDied","Data":"2487d6ce6e5ec41746867cddb28a5148a1879bd0e4ee46703049fef804dba1d9"}
Nov 02 13:08:51.928273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.928232    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2487d6ce6e5ec41746867cddb28a5148a1879bd0e4ee46703049fef804dba1d9"
Nov 02 13:08:51.929907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.929882    5879 generic.go:356] "Generic (PLEG): container finished" podID="556a66bf-9cc7-49ec-b9b5-066c7976e718" containerID="c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543" exitCode=0
Nov 02 13:08:51.930071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.929920    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"
Nov 02 13:08:51.930071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.929962    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw" event={"ID":"556a66bf-9cc7-49ec-b9b5-066c7976e718","Type":"ContainerDied","Data":"c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543"}
Nov 02 13:08:51.930071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.930059    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw" event={"ID":"556a66bf-9cc7-49ec-b9b5-066c7976e718","Type":"ContainerDied","Data":"21f70096e189a8f51fbee3520658b88c8112c12b201c0fc1cf3fd3f5047951e4"}
Nov 02 13:08:51.930251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.930093    5879 scope.go:122] "RemoveContainer" containerID="c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543"
Nov 02 13:08:51.947221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.946496    5879 scope.go:122] "RemoveContainer" containerID="c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543"
Nov 02 13:08:51.947221 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:51.946984    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543\": container with ID starting with c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543 not found: ID does not exist" containerID="c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543"
Nov 02 13:08:51.947221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:51.947060    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543"} err="failed to get container status \"c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543\": rpc error: code = NotFound desc = could not find container \"c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543\": container with ID starting with c567c993d1c132234e958f1e07ac1d208e79fc4fb70a11288350e957541a7543 not found: ID does not exist"
Nov 02 13:08:52.036447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:52.036411    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"]
Nov 02 13:08:52.041621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:52.041582    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-25f7a6/payload-processing-external-model-e2e-extmod-a-25f7a6-78f8dqhvw"]
Nov 02 13:08:53.148169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:53.148087    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:53.390649 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:53.390595    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:54.021015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:54.020971    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="556a66bf-9cc7-49ec-b9b5-066c7976e718" path="/var/lib/kubelet/pods/556a66bf-9cc7-49ec-b9b5-066c7976e718/volumes"
Nov 02 13:08:55.084166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:55.084134    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:56.883720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:56.883678    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:56.928802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:56.928765    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c" podUID="4c13cde5-9860-4b39-ae37-c62d6ce8cca5" containerName="payload-processing" containerID="cri-o://cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8" gracePeriod=25
Nov 02 13:08:57.206641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.206617    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:08:57.306733 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.306698    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-kube-api-access-9m9c7\" (UniqueName: \"kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-kube-api-access-9m9c7\") pod \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") "
Nov 02 13:08:57.306886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.306836    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-plugins-config-volume\") pod \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") "
Nov 02 13:08:57.306976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.306965    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-routing-overlay-volume\") pod \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") "
Nov 02 13:08:57.307060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.306988    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-provider-credentials\") pod \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\" (UID: \"4c13cde5-9860-4b39-ae37-c62d6ce8cca5\") "
Nov 02 13:08:57.307326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.307301    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-plugins-config-volume" pod "4c13cde5-9860-4b39-ae37-c62d6ce8cca5" (UID: "4c13cde5-9860-4b39-ae37-c62d6ce8cca5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:57.307404 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.307318    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-routing-overlay-volume" pod "4c13cde5-9860-4b39-ae37-c62d6ce8cca5" (UID: "4c13cde5-9860-4b39-ae37-c62d6ce8cca5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:08:57.307671 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.307651    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:57.307671 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.307670    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:57.309355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.309330    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-kube-api-access-9m9c7" pod "4c13cde5-9860-4b39-ae37-c62d6ce8cca5" (UID: "4c13cde5-9860-4b39-ae37-c62d6ce8cca5"). InnerVolumeSpecName "kube-api-access-9m9c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:57.309468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.309425    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-provider-credentials" pod "4c13cde5-9860-4b39-ae37-c62d6ce8cca5" (UID: "4c13cde5-9860-4b39-ae37-c62d6ce8cca5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:08:57.409697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.409661    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m9c7\" (UniqueName: \"kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-kube-api-access-9m9c7\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:57.409697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.409693    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c13cde5-9860-4b39-ae37-c62d6ce8cca5-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:08:57.981642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.981601    5879 generic.go:356] "Generic (PLEG): container finished" podID="4c13cde5-9860-4b39-ae37-c62d6ce8cca5" containerID="cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8" exitCode=0
Nov 02 13:08:57.982165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.981651    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"
Nov 02 13:08:57.982165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.981686    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c" event={"ID":"4c13cde5-9860-4b39-ae37-c62d6ce8cca5","Type":"ContainerDied","Data":"cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8"}
Nov 02 13:08:57.982165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.981760    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c" event={"ID":"4c13cde5-9860-4b39-ae37-c62d6ce8cca5","Type":"ContainerDied","Data":"e9f4ffb86bdb7f85c35a4980dfad1cdca48724dd2dad108ff60ef28100688e2c"}
Nov 02 13:08:57.982165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.981793    5879 scope.go:122] "RemoveContainer" containerID="cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8"
Nov 02 13:08:57.993773 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.993756    5879 scope.go:122] "RemoveContainer" containerID="cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8"
Nov 02 13:08:57.994052 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:08:57.994034    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8\": container with ID starting with cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8 not found: ID does not exist" containerID="cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8"
Nov 02 13:08:57.994164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:57.994060    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8"} err="failed to get container status \"cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8\": rpc error: code = NotFound desc = could not find container \"cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8\": container with ID starting with cfc4f3e47ae7fa4356cafaeff084f9985591f3ff298eb1a67444d9c179fb30e8 not found: ID does not exist"
Nov 02 13:08:58.074066 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:58.074026    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"]
Nov 02 13:08:58.087411 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:58.087378    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-6dd69hd4c"]
Nov 02 13:08:58.977587 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:58.977550    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:08:59.102968 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.102928    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"]
Nov 02 13:08:59.103813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.103786    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e0e7e43d-334c-4367-8642-5669d5df4c8e" containerName="revoke-keys"
Nov 02 13:08:59.103813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.103811    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e7e43d-334c-4367-8642-5669d5df4c8e" containerName="revoke-keys"
Nov 02 13:08:59.104002 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.103876    5879 cpu_manager.go:414] "containerMap: removing container" podUID="556a66bf-9cc7-49ec-b9b5-066c7976e718" containerName="payload-processing"
Nov 02 13:08:59.104002 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.103884    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556a66bf-9cc7-49ec-b9b5-066c7976e718" containerName="payload-processing"
Nov 02 13:08:59.104002 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.103902    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4c13cde5-9860-4b39-ae37-c62d6ce8cca5" containerName="payload-processing"
Nov 02 13:08:59.104002 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.103910    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c13cde5-9860-4b39-ae37-c62d6ce8cca5" containerName="payload-processing"
Nov 02 13:08:59.104182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.104167    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="556a66bf-9cc7-49ec-b9b5-066c7976e718" containerName="payload-processing"
Nov 02 13:08:59.104226 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.104187    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c13cde5-9860-4b39-ae37-c62d6ce8cca5" containerName="payload-processing"
Nov 02 13:08:59.104226 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.104207    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e7e43d-334c-4367-8642-5669d5df4c8e" containerName="revoke-keys"
Nov 02 13:08:59.108569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.108552    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:08:59.129387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.129357    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"]
Nov 02 13:08:59.136477 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.136452    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:08:59.136724 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.136699    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:08:59.236005 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.235899    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08689e60-6vr6m\" (UID: \"be954e1c-33e9-44cf-9365-c88f37d39856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:08:59.236174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.236114    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b7jw\" (UniqueName: \"kubernetes.io/projected/be954e1c-33e9-44cf-9365-c88f37d39856-kube-api-access-8b7jw\") pod \"maas-api-revoke-sub-08689e60-6vr6m\" (UID: \"be954e1c-33e9-44cf-9365-c88f37d39856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:08:59.236246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.236228    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08689e60-6vr6m\" (UID: \"be954e1c-33e9-44cf-9365-c88f37d39856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:08:59.338139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.338101    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b7jw\" (UniqueName: \"kubernetes.io/projected/be954e1c-33e9-44cf-9365-c88f37d39856-kube-api-access-8b7jw\") pod \"maas-api-revoke-sub-08689e60-6vr6m\" (UID: \"be954e1c-33e9-44cf-9365-c88f37d39856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:08:59.338338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.338197    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08689e60-6vr6m\" (UID: \"be954e1c-33e9-44cf-9365-c88f37d39856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:08:59.338338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.338280    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08689e60-6vr6m\" (UID: \"be954e1c-33e9-44cf-9365-c88f37d39856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:08:59.338917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.338898    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08689e60-6vr6m\" (UID: \"be954e1c-33e9-44cf-9365-c88f37d39856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:08:59.340782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.340757    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08689e60-6vr6m\" (UID: \"be954e1c-33e9-44cf-9365-c88f37d39856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:08:59.363215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.363183    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b7jw\" (UniqueName: \"kubernetes.io/projected/be954e1c-33e9-44cf-9365-c88f37d39856-kube-api-access-8b7jw\") pod \"maas-api-revoke-sub-08689e60-6vr6m\" (UID: \"be954e1c-33e9-44cf-9365-c88f37d39856\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:08:59.408053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.408016    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"]
Nov 02 13:08:59.412782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.412758    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:08:59.425164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.425132    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:08:59.439652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.439620    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"]
Nov 02 13:08:59.554132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.553942    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59beaad8-vrlf2\" (UID: \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:08:59.555776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.555200    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdfvx\" (UniqueName: \"kubernetes.io/projected/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-kube-api-access-kdfvx\") pod \"maas-api-revoke-sub-59beaad8-vrlf2\" (UID: \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:08:59.555776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.555253    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59beaad8-vrlf2\" (UID: \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:08:59.657186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.657149    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59beaad8-vrlf2\" (UID: \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:08:59.657383 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.657257    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdfvx\" (UniqueName: \"kubernetes.io/projected/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-kube-api-access-kdfvx\") pod \"maas-api-revoke-sub-59beaad8-vrlf2\" (UID: \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:08:59.657383 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.657290    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59beaad8-vrlf2\" (UID: \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:08:59.658034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.658007    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59beaad8-vrlf2\" (UID: \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:08:59.659875 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.659853    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59beaad8-vrlf2\" (UID: \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:08:59.687611 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.687578    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdfvx\" (UniqueName: \"kubernetes.io/projected/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-kube-api-access-kdfvx\") pod \"maas-api-revoke-sub-59beaad8-vrlf2\" (UID: \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:08:59.729471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.729441    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:08:59.761126 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:08:59.761093    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe954e1c_33e9_44cf_9365_c88f37d39856.slice/crio-1021163a07c8f8c2e994e3f8ca225a1fbc71e8c24fd671d6030a4a2b6d50a102 WatchSource:0}: Error finding container 1021163a07c8f8c2e994e3f8ca225a1fbc71e8c24fd671d6030a4a2b6d50a102: Status 404 returned error can't find the container with id 1021163a07c8f8c2e994e3f8ca225a1fbc71e8c24fd671d6030a4a2b6d50a102
Nov 02 13:08:59.762707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:08:59.762679    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"]
Nov 02 13:09:00.004555 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:00.004511    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m" event={"ID":"be954e1c-33e9-44cf-9365-c88f37d39856","Type":"ContainerStarted","Data":"1021163a07c8f8c2e994e3f8ca225a1fbc71e8c24fd671d6030a4a2b6d50a102"}
Nov 02 13:09:00.019821 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:00.019748    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c13cde5-9860-4b39-ae37-c62d6ce8cca5" path="/var/lib/kubelet/pods/4c13cde5-9860-4b39-ae37-c62d6ce8cca5/volumes"
Nov 02 13:09:00.052346 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:00.052314    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa80df6_2b05_4ff5_b27b_0cf4a8958848.slice/crio-cc4b592f3828e83f8f8c701849d34201bf1977b81ca32de16a792d230997d3e0 WatchSource:0}: Error finding container cc4b592f3828e83f8f8c701849d34201bf1977b81ca32de16a792d230997d3e0: Status 404 returned error can't find the container with id cc4b592f3828e83f8f8c701849d34201bf1977b81ca32de16a792d230997d3e0
Nov 02 13:09:00.053032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:00.053005    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"]
Nov 02 13:09:00.504467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:00.504437    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:00.640319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:00.640283    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:00.977580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:00.977549    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:01.022767 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.022729    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2" event={"ID":"aaa80df6-2b05-4ff5-b27b-0cf4a8958848","Type":"ContainerStarted","Data":"6d8a60214ee5f0000ed27f14658ebbcb3955716993236f0580f5f90d82f85ac6"}
Nov 02 13:09:01.023032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.022819    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2" event={"ID":"aaa80df6-2b05-4ff5-b27b-0cf4a8958848","Type":"ContainerStarted","Data":"cc4b592f3828e83f8f8c701849d34201bf1977b81ca32de16a792d230997d3e0"}
Nov 02 13:09:01.025037 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.024974    5879 generic.go:356] "Generic (PLEG): container finished" podID="be954e1c-33e9-44cf-9365-c88f37d39856" containerID="9f096cb35bb6353626d186628134ed3969a796f8bc165fc83506a2d3f2626ec8" exitCode=0
Nov 02 13:09:01.025191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.025170    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m" event={"ID":"be954e1c-33e9-44cf-9365-c88f37d39856","Type":"ContainerDied","Data":"9f096cb35bb6353626d186628134ed3969a796f8bc165fc83506a2d3f2626ec8"}
Nov 02 13:09:01.089849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.089739    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2" podStartSLOduration=2.089721852 podStartE2EDuration="2.089721852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:01.087110976 +0000 UTC m=+2487.193813827" watchObservedRunningTime="2026-11-02 13:09:01.089721852 +0000 UTC m=+2487.196424703"
Nov 02 13:09:01.367603 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.367568    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:01.482945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.482910    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:01.636258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.636147    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"]
Nov 02 13:09:01.642489 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.642460    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:01.683018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.680464    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"]
Nov 02 13:09:01.690257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.690216    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxr4v\" (UniqueName: \"kubernetes.io/projected/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-kube-api-access-xxr4v\") pod \"maas-api-revoke-sub-2973c15c-5ktrt\" (UID: \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:01.690462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.690276    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2973c15c-5ktrt\" (UID: \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:01.690462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.690438    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2973c15c-5ktrt\" (UID: \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:01.792476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.792435    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2973c15c-5ktrt\" (UID: \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:01.792781 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.792761    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxr4v\" (UniqueName: \"kubernetes.io/projected/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-kube-api-access-xxr4v\") pod \"maas-api-revoke-sub-2973c15c-5ktrt\" (UID: \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:01.792835 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.792808    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2973c15c-5ktrt\" (UID: \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:01.793677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.793648    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2973c15c-5ktrt\" (UID: \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:01.795293 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.795273    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2973c15c-5ktrt\" (UID: \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:01.817742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.817715    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxr4v\" (UniqueName: \"kubernetes.io/projected/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-kube-api-access-xxr4v\") pod \"maas-api-revoke-sub-2973c15c-5ktrt\" (UID: \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:01.960434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:01.960350    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:02.042254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.042209    5879 generic.go:356] "Generic (PLEG): container finished" podID="aaa80df6-2b05-4ff5-b27b-0cf4a8958848" containerID="6d8a60214ee5f0000ed27f14658ebbcb3955716993236f0580f5f90d82f85ac6" exitCode=0
Nov 02 13:09:02.043446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.042920    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2" event={"ID":"aaa80df6-2b05-4ff5-b27b-0cf4a8958848","Type":"ContainerDied","Data":"6d8a60214ee5f0000ed27f14658ebbcb3955716993236f0580f5f90d82f85ac6"}
Nov 02 13:09:02.218542 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.218474    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:09:02.286388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.286338    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"]
Nov 02 13:09:02.302132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.300789    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-openshift-ca\") pod \"be954e1c-33e9-44cf-9365-c88f37d39856\" (UID: \"be954e1c-33e9-44cf-9365-c88f37d39856\") "
Nov 02 13:09:02.302132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.300960    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be954e1c-33e9-44cf-9365-c88f37d39856-kube-api-access-8b7jw\" (UniqueName: \"kubernetes.io/projected/be954e1c-33e9-44cf-9365-c88f37d39856-kube-api-access-8b7jw\") pod \"be954e1c-33e9-44cf-9365-c88f37d39856\" (UID: \"be954e1c-33e9-44cf-9365-c88f37d39856\") "
Nov 02 13:09:02.302132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.301073    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-xks-ca\") pod \"be954e1c-33e9-44cf-9365-c88f37d39856\" (UID: \"be954e1c-33e9-44cf-9365-c88f37d39856\") "
Nov 02 13:09:02.306608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.306577    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-openshift-ca" pod "be954e1c-33e9-44cf-9365-c88f37d39856" (UID: "be954e1c-33e9-44cf-9365-c88f37d39856"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:02.309438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.309407    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be954e1c-33e9-44cf-9365-c88f37d39856-kube-api-access-8b7jw" pod "be954e1c-33e9-44cf-9365-c88f37d39856" (UID: "be954e1c-33e9-44cf-9365-c88f37d39856"). InnerVolumeSpecName "kube-api-access-8b7jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:02.310405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.310371    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-xks-ca" pod "be954e1c-33e9-44cf-9365-c88f37d39856" (UID: "be954e1c-33e9-44cf-9365-c88f37d39856"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:09:02.403046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.402981    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:02.403046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.403039    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be954e1c-33e9-44cf-9365-c88f37d39856-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:02.403046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.403052    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b7jw\" (UniqueName: \"kubernetes.io/projected/be954e1c-33e9-44cf-9365-c88f37d39856-kube-api-access-8b7jw\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:02.792205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.792168    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:02.948897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:02.948854    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:03.043747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.040970    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"]
Nov 02 13:09:03.043747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.042280    5879 cpu_manager.go:414] "containerMap: removing container" podUID="be954e1c-33e9-44cf-9365-c88f37d39856" containerName="revoke-keys"
Nov 02 13:09:03.043747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.042305    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be954e1c-33e9-44cf-9365-c88f37d39856" containerName="revoke-keys"
Nov 02 13:09:03.043747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.042500    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="be954e1c-33e9-44cf-9365-c88f37d39856" containerName="revoke-keys"
Nov 02 13:09:03.062040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.056703    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.095298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.082203    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt" event={"ID":"caff3b86-d5ba-40bf-a7d2-9192f4d509d5","Type":"ContainerStarted","Data":"948c1160154dcba48b4c3e28b947651d75f9d7791839612ff4b988704bd4d1d5"}
Nov 02 13:09:03.095298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.082276    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt" event={"ID":"caff3b86-d5ba-40bf-a7d2-9192f4d509d5","Type":"ContainerStarted","Data":"54abefb1d81c70ca559df295f6c0fdd58b3327c3ecae8152ecb79fd6d4ab1db5"}
Nov 02 13:09:03.095298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.093498    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-ef5efe\"/\"agc-cred-ad8f1188522d\""
Nov 02 13:09:03.095298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.094563    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"
Nov 02 13:09:03.095298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.095050    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m" event={"ID":"be954e1c-33e9-44cf-9365-c88f37d39856","Type":"ContainerDied","Data":"1021163a07c8f8c2e994e3f8ca225a1fbc71e8c24fd671d6030a4a2b6d50a102"}
Nov 02 13:09:03.095298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.095109    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1021163a07c8f8c2e994e3f8ca225a1fbc71e8c24fd671d6030a4a2b6d50a102"
Nov 02 13:09:03.107477 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.107446    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"]
Nov 02 13:09:03.114973 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.114797    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.115144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.115108    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.115225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.115178    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.115312 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.115284    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pzvh\" (UniqueName: \"kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-kube-api-access-2pzvh\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.150095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.150036    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt" podStartSLOduration=2.149981198 podStartE2EDuration="2.149981198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:03.149158381 +0000 UTC m=+2489.255861233" watchObservedRunningTime="2026-11-02 13:09:03.149981198 +0000 UTC m=+2489.256684051"
Nov 02 13:09:03.218067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.216224    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.218067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.216328    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.218067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.216399    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pzvh\" (UniqueName: \"kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-kube-api-access-2pzvh\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.218067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.216564    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.218067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.217448    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.218067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.217924    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.232038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.226950    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.250751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.248590    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:03.251965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.251934    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pzvh\" (UniqueName: \"kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-kube-api-access-2pzvh\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.382060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.382027    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:09:03.419264 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.419229    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-kube-api-access-kdfvx\" (UniqueName: \"kubernetes.io/projected/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-kube-api-access-kdfvx\") pod \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\" (UID: \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\") "
Nov 02 13:09:03.419451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.419322    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-xks-ca\") pod \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\" (UID: \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\") "
Nov 02 13:09:03.419451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.419379    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-openshift-ca\") pod \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\" (UID: \"aaa80df6-2b05-4ff5-b27b-0cf4a8958848\") "
Nov 02 13:09:03.419764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.419737    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-openshift-ca" pod "aaa80df6-2b05-4ff5-b27b-0cf4a8958848" (UID: "aaa80df6-2b05-4ff5-b27b-0cf4a8958848"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:03.420213 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.420189    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:03.421615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.421592    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-xks-ca" pod "aaa80df6-2b05-4ff5-b27b-0cf4a8958848" (UID: "aaa80df6-2b05-4ff5-b27b-0cf4a8958848"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:09:03.421731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.421699    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-kube-api-access-kdfvx" pod "aaa80df6-2b05-4ff5-b27b-0cf4a8958848" (UID: "aaa80df6-2b05-4ff5-b27b-0cf4a8958848"). InnerVolumeSpecName "kube-api-access-kdfvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:03.424850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.424829    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:03.522070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.522034    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:03.522070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.522061    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdfvx\" (UniqueName: \"kubernetes.io/projected/aaa80df6-2b05-4ff5-b27b-0cf4a8958848-kube-api-access-kdfvx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:03.747120 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:03.747092    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9123562e_d96a_4a86_9764_519429f8f5f6.slice/crio-bfcb31c112635e680286d3485a32110c03c3e82ba496a960bf21c9bfb162eff1 WatchSource:0}: Error finding container bfcb31c112635e680286d3485a32110c03c3e82ba496a960bf21c9bfb162eff1: Status 404 returned error can't find the container with id bfcb31c112635e680286d3485a32110c03c3e82ba496a960bf21c9bfb162eff1
Nov 02 13:09:03.747235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:03.747214    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"]
Nov 02 13:09:04.029414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.029389    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"]
Nov 02 13:09:04.030510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.030485    5879 cpu_manager.go:414] "containerMap: removing container" podUID="aaa80df6-2b05-4ff5-b27b-0cf4a8958848" containerName="revoke-keys"
Nov 02 13:09:04.030510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.030509    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa80df6-2b05-4ff5-b27b-0cf4a8958848" containerName="revoke-keys"
Nov 02 13:09:04.030702 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.030682    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa80df6-2b05-4ff5-b27b-0cf4a8958848" containerName="revoke-keys"
Nov 02 13:09:04.034871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.034852    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.051374 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.051332    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-8e7fc4b63cd8\""
Nov 02 13:09:04.053922 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.053897    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"]
Nov 02 13:09:04.104309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.104276    5879 generic.go:356] "Generic (PLEG): container finished" podID="caff3b86-d5ba-40bf-a7d2-9192f4d509d5" containerID="948c1160154dcba48b4c3e28b947651d75f9d7791839612ff4b988704bd4d1d5" exitCode=0
Nov 02 13:09:04.104487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.104370    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt" event={"ID":"caff3b86-d5ba-40bf-a7d2-9192f4d509d5","Type":"ContainerDied","Data":"948c1160154dcba48b4c3e28b947651d75f9d7791839612ff4b988704bd4d1d5"}
Nov 02 13:09:04.106126 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.106097    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l" event={"ID":"9123562e-d96a-4a86-9764-519429f8f5f6","Type":"ContainerStarted","Data":"828b5e48463f0cf3f1df8b79f51b8c2acac45558a2f8d277012edc02966f5da8"}
Nov 02 13:09:04.106274 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.106141    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l" event={"ID":"9123562e-d96a-4a86-9764-519429f8f5f6","Type":"ContainerStarted","Data":"bfcb31c112635e680286d3485a32110c03c3e82ba496a960bf21c9bfb162eff1"}
Nov 02 13:09:04.106274 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.106183    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:04.107468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.107448    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"
Nov 02 13:09:04.107468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.107454    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2" event={"ID":"aaa80df6-2b05-4ff5-b27b-0cf4a8958848","Type":"ContainerDied","Data":"cc4b592f3828e83f8f8c701849d34201bf1977b81ca32de16a792d230997d3e0"}
Nov 02 13:09:04.107623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.107495    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc4b592f3828e83f8f8c701849d34201bf1977b81ca32de16a792d230997d3e0"
Nov 02 13:09:04.130219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.130187    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-provider-credentials\") pod \"payload-processing-external-model-9588b4f4b-jbz64\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") " pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.130371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.130311    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92tb2\" (UniqueName: \"kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-kube-api-access-92tb2\") pod \"payload-processing-external-model-9588b4f4b-jbz64\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") " pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.130926 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.130602    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-plugins-config-volume\") pod \"payload-processing-external-model-9588b4f4b-jbz64\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") " pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.130926 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.130827    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-routing-overlay-volume\") pod \"payload-processing-external-model-9588b4f4b-jbz64\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") " pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.193159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.193092    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l" podStartSLOduration=1.193071725 podStartE2EDuration="1.193071725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:04.189894896 +0000 UTC m=+2490.296597746" watchObservedRunningTime="2026-11-02 13:09:04.193071725 +0000 UTC m=+2490.299774578"
Nov 02 13:09:04.232104 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.232063    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-plugins-config-volume\") pod \"payload-processing-external-model-9588b4f4b-jbz64\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") " pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.232320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.232215    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-routing-overlay-volume\") pod \"payload-processing-external-model-9588b4f4b-jbz64\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") " pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.232320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.232239    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-provider-credentials\") pod \"payload-processing-external-model-9588b4f4b-jbz64\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") " pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.232320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.232310    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92tb2\" (UniqueName: \"kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-kube-api-access-92tb2\") pod \"payload-processing-external-model-9588b4f4b-jbz64\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") " pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.232948 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.232916    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-routing-overlay-volume\") pod \"payload-processing-external-model-9588b4f4b-jbz64\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") " pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.233433 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.233414    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-plugins-config-volume\") pod \"payload-processing-external-model-9588b4f4b-jbz64\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") " pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.235180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.235149    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-provider-credentials\") pod \"payload-processing-external-model-9588b4f4b-jbz64\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") " pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.257561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.257535    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92tb2\" (UniqueName: \"kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-kube-api-access-92tb2\") pod \"payload-processing-external-model-9588b4f4b-jbz64\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") " pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.350058 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.349940    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:04.659737 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:04.659693    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3679e0cc_4500_409a_860c_2dee524e4187.slice/crio-fa6289b9febc5593e81078d8a90aa3efaca22f4e662adf58c460c9cc44f0dcd8 WatchSource:0}: Error finding container fa6289b9febc5593e81078d8a90aa3efaca22f4e662adf58c460c9cc44f0dcd8: Status 404 returned error can't find the container with id fa6289b9febc5593e81078d8a90aa3efaca22f4e662adf58c460c9cc44f0dcd8
Nov 02 13:09:04.661562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:04.661493    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"]
Nov 02 13:09:05.119070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.119033    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64" event={"ID":"3679e0cc-4500-409a-860c-2dee524e4187","Type":"ContainerStarted","Data":"f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633"}
Nov 02 13:09:05.119560 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.119105    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64" event={"ID":"3679e0cc-4500-409a-860c-2dee524e4187","Type":"ContainerStarted","Data":"fa6289b9febc5593e81078d8a90aa3efaca22f4e662adf58c460c9cc44f0dcd8"}
Nov 02 13:09:05.119560 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.119338    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:05.164355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.164289    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64" podStartSLOduration=1.16427471 podStartE2EDuration="1.16427471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:05.162341447 +0000 UTC m=+2491.269044298" watchObservedRunningTime="2026-11-02 13:09:05.16427471 +0000 UTC m=+2491.270977560"
Nov 02 13:09:05.188010 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.187957    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:05.312778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.312753    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:05.348271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.348234    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-kube-api-access-xxr4v\" (UniqueName: \"kubernetes.io/projected/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-kube-api-access-xxr4v\") pod \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\" (UID: \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\") "
Nov 02 13:09:05.348499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.348322    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-openshift-ca\") pod \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\" (UID: \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\") "
Nov 02 13:09:05.348499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.348423    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-xks-ca\") pod \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\" (UID: \"caff3b86-d5ba-40bf-a7d2-9192f4d509d5\") "
Nov 02 13:09:05.348738 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.348713    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-openshift-ca" pod "caff3b86-d5ba-40bf-a7d2-9192f4d509d5" (UID: "caff3b86-d5ba-40bf-a7d2-9192f4d509d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:05.349092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.349072    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:05.350656 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.350637    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-xks-ca" pod "caff3b86-d5ba-40bf-a7d2-9192f4d509d5" (UID: "caff3b86-d5ba-40bf-a7d2-9192f4d509d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:09:05.350734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.350702    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-kube-api-access-xxr4v" pod "caff3b86-d5ba-40bf-a7d2-9192f4d509d5" (UID: "caff3b86-d5ba-40bf-a7d2-9192f4d509d5"). InnerVolumeSpecName "kube-api-access-xxr4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:05.450406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.450321    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxr4v\" (UniqueName: \"kubernetes.io/projected/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-kube-api-access-xxr4v\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:05.450406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:05.450349    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caff3b86-d5ba-40bf-a7d2-9192f4d509d5-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:06.128621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:06.128591    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"
Nov 02 13:09:06.129036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:06.128622    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt" event={"ID":"caff3b86-d5ba-40bf-a7d2-9192f4d509d5","Type":"ContainerDied","Data":"54abefb1d81c70ca559df295f6c0fdd58b3327c3ecae8152ecb79fd6d4ab1db5"}
Nov 02 13:09:06.129036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:06.128667    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54abefb1d81c70ca559df295f6c0fdd58b3327c3ecae8152ecb79fd6d4ab1db5"
Nov 02 13:09:06.956776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:06.956746    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:08.771239 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:08.771201    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:09.322867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:09.322835    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:09.810035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:09.810002    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"]
Nov 02 13:09:09.810856 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:09.810831    5879 cpu_manager.go:414] "containerMap: removing container" podUID="caff3b86-d5ba-40bf-a7d2-9192f4d509d5" containerName="revoke-keys"
Nov 02 13:09:09.810856 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:09.810851    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caff3b86-d5ba-40bf-a7d2-9192f4d509d5" containerName="revoke-keys"
Nov 02 13:09:09.811074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:09.811061    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="caff3b86-d5ba-40bf-a7d2-9192f4d509d5" containerName="revoke-keys"
Nov 02 13:09:09.815456 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:09.815439    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:09.836402 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:09.836371    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"]
Nov 02 13:09:09.845037 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:09.844801    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:09:09.845037 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:09.844861    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:09:09.913866 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:09.913818    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2cd78d-8135-471c-b1de-176466518666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a11a10-gxjgk\" (UID: \"5d2cd78d-8135-471c-b1de-176466518666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:09.914099 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:09.914027    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2cd78d-8135-471c-b1de-176466518666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a11a10-gxjgk\" (UID: \"5d2cd78d-8135-471c-b1de-176466518666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:09.914173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:09.914155    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlpf4\" (UniqueName: \"kubernetes.io/projected/5d2cd78d-8135-471c-b1de-176466518666-kube-api-access-nlpf4\") pod \"maas-api-revoke-sub-93a11a10-gxjgk\" (UID: \"5d2cd78d-8135-471c-b1de-176466518666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:10.015500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:10.015468    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlpf4\" (UniqueName: \"kubernetes.io/projected/5d2cd78d-8135-471c-b1de-176466518666-kube-api-access-nlpf4\") pod \"maas-api-revoke-sub-93a11a10-gxjgk\" (UID: \"5d2cd78d-8135-471c-b1de-176466518666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:10.015689 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:10.015582    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2cd78d-8135-471c-b1de-176466518666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a11a10-gxjgk\" (UID: \"5d2cd78d-8135-471c-b1de-176466518666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:10.015742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:10.015699    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2cd78d-8135-471c-b1de-176466518666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a11a10-gxjgk\" (UID: \"5d2cd78d-8135-471c-b1de-176466518666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:10.016382 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:10.016357    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2cd78d-8135-471c-b1de-176466518666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a11a10-gxjgk\" (UID: \"5d2cd78d-8135-471c-b1de-176466518666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:10.018286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:10.018266    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2cd78d-8135-471c-b1de-176466518666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a11a10-gxjgk\" (UID: \"5d2cd78d-8135-471c-b1de-176466518666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:10.040842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:10.040811    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlpf4\" (UniqueName: \"kubernetes.io/projected/5d2cd78d-8135-471c-b1de-176466518666-kube-api-access-nlpf4\") pod \"maas-api-revoke-sub-93a11a10-gxjgk\" (UID: \"5d2cd78d-8135-471c-b1de-176466518666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:10.130645 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:10.130610    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:10.444534 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:10.444505    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2cd78d_8135_471c_b1de_176466518666.slice/crio-57b62d0b257890b364cdcdf95c917173f96c3ac84a9049c436f295f17edd37cc WatchSource:0}: Error finding container 57b62d0b257890b364cdcdf95c917173f96c3ac84a9049c436f295f17edd37cc: Status 404 returned error can't find the container with id 57b62d0b257890b364cdcdf95c917173f96c3ac84a9049c436f295f17edd37cc
Nov 02 13:09:10.445363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:10.445340    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"]
Nov 02 13:09:10.549478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:10.549442    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:10.671242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:10.671202    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:10.790131 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:10.790101    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:11.175657 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.175623    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk" event={"ID":"5d2cd78d-8135-471c-b1de-176466518666","Type":"ContainerStarted","Data":"c85f42313c65909f02188262e16d9eb984eaf2add83d64c6e40fbf3015de0303"}
Nov 02 13:09:11.176158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.175685    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk" event={"ID":"5d2cd78d-8135-471c-b1de-176466518666","Type":"ContainerStarted","Data":"57b62d0b257890b364cdcdf95c917173f96c3ac84a9049c436f295f17edd37cc"}
Nov 02 13:09:11.210159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.210126    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"]
Nov 02 13:09:11.215084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.215056    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.219904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.219839    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk" podStartSLOduration=2.219821379 podStartE2EDuration="2.219821379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:11.218218588 +0000 UTC m=+2497.324921443" watchObservedRunningTime="2026-11-02 13:09:11.219821379 +0000 UTC m=+2497.326524232"
Nov 02 13:09:11.235486 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.235451    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"]
Nov 02 13:09:11.317078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.317042    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"]
Nov 02 13:09:11.322041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.322018    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:11.336517 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.336481    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.336761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.336737    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.336941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.336917    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c66t4\" (UniqueName: \"kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-kube-api-access-c66t4\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.337045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.336986    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.340986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.340962    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"]
Nov 02 13:09:11.445069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.438343    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056e1508-5e60-4712-8765-e871d3339852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-152184f0-d9pkf\" (UID: \"056e1508-5e60-4712-8765-e871d3339852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:11.445069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.438472    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056e1508-5e60-4712-8765-e871d3339852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-152184f0-d9pkf\" (UID: \"056e1508-5e60-4712-8765-e871d3339852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:11.445069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.438541    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.445069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.438594    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs7pk\" (UniqueName: \"kubernetes.io/projected/056e1508-5e60-4712-8765-e871d3339852-kube-api-access-qs7pk\") pod \"maas-api-revoke-sub-152184f0-d9pkf\" (UID: \"056e1508-5e60-4712-8765-e871d3339852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:11.445069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.438682    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c66t4\" (UniqueName: \"kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-kube-api-access-c66t4\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.445069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.438725    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.445069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.438874    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.445069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.441728    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.445069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.441824    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.474707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.470747    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.480435 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.480403    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c66t4\" (UniqueName: \"kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-kube-api-access-c66t4\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.535041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.534981    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:11.540965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.540936    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056e1508-5e60-4712-8765-e871d3339852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-152184f0-d9pkf\" (UID: \"056e1508-5e60-4712-8765-e871d3339852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:11.541110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.541090    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056e1508-5e60-4712-8765-e871d3339852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-152184f0-d9pkf\" (UID: \"056e1508-5e60-4712-8765-e871d3339852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:11.541204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.541179    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs7pk\" (UniqueName: \"kubernetes.io/projected/056e1508-5e60-4712-8765-e871d3339852-kube-api-access-qs7pk\") pod \"maas-api-revoke-sub-152184f0-d9pkf\" (UID: \"056e1508-5e60-4712-8765-e871d3339852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:11.541770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.541748    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056e1508-5e60-4712-8765-e871d3339852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-152184f0-d9pkf\" (UID: \"056e1508-5e60-4712-8765-e871d3339852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:11.543707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.543685    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056e1508-5e60-4712-8765-e871d3339852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-152184f0-d9pkf\" (UID: \"056e1508-5e60-4712-8765-e871d3339852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:11.566430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.566402    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs7pk\" (UniqueName: \"kubernetes.io/projected/056e1508-5e60-4712-8765-e871d3339852-kube-api-access-qs7pk\") pod \"maas-api-revoke-sub-152184f0-d9pkf\" (UID: \"056e1508-5e60-4712-8765-e871d3339852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:11.638325 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.638285    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:11.846578 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:11.846549    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8b12d6_622a_49d1_bd56_41a22e7f4415.slice/crio-186412b5d533f40569626a82395ecd966481619551cb8378630a2eb29ab94731 WatchSource:0}: Error finding container 186412b5d533f40569626a82395ecd966481619551cb8378630a2eb29ab94731: Status 404 returned error can't find the container with id 186412b5d533f40569626a82395ecd966481619551cb8378630a2eb29ab94731
Nov 02 13:09:11.849510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.849485    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"]
Nov 02 13:09:11.956014 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:11.955925    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056e1508_5e60_4712_8765_e871d3339852.slice/crio-aee3420c7c39d0dedf4457aa59528022d97948b9081d37108438edf14ee70bef WatchSource:0}: Error finding container aee3420c7c39d0dedf4457aa59528022d97948b9081d37108438edf14ee70bef: Status 404 returned error can't find the container with id aee3420c7c39d0dedf4457aa59528022d97948b9081d37108438edf14ee70bef
Nov 02 13:09:11.957279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:11.957253    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"]
Nov 02 13:09:12.187168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:12.187129    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf" event={"ID":"056e1508-5e60-4712-8765-e871d3339852","Type":"ContainerStarted","Data":"aee3420c7c39d0dedf4457aa59528022d97948b9081d37108438edf14ee70bef"}
Nov 02 13:09:12.189839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:12.189809    5879 generic.go:356] "Generic (PLEG): container finished" podID="5d2cd78d-8135-471c-b1de-176466518666" containerID="c85f42313c65909f02188262e16d9eb984eaf2add83d64c6e40fbf3015de0303" exitCode=0
Nov 02 13:09:12.190017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:12.189881    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk" event={"ID":"5d2cd78d-8135-471c-b1de-176466518666","Type":"ContainerDied","Data":"c85f42313c65909f02188262e16d9eb984eaf2add83d64c6e40fbf3015de0303"}
Nov 02 13:09:12.191612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:12.191590    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk" event={"ID":"0e8b12d6-622a-49d1-bd56-41a22e7f4415","Type":"ContainerStarted","Data":"57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c"}
Nov 02 13:09:12.191708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:12.191629    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk" event={"ID":"0e8b12d6-622a-49d1-bd56-41a22e7f4415","Type":"ContainerStarted","Data":"186412b5d533f40569626a82395ecd966481619551cb8378630a2eb29ab94731"}
Nov 02 13:09:12.191708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:12.191651    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:12.325647 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:12.325546    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk" podStartSLOduration=1.325529942 podStartE2EDuration="1.325529942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:12.322891554 +0000 UTC m=+2498.429594404" watchObservedRunningTime="2026-11-02 13:09:12.325529942 +0000 UTC m=+2498.432232829"
Nov 02 13:09:12.400899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:12.400869    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:13.202059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:13.201983    5879 generic.go:356] "Generic (PLEG): container finished" podID="056e1508-5e60-4712-8765-e871d3339852" containerID="ebe5152b5dc9fa561079258f749e376aec7a02c81b8bee8e9df63895b4c7a139" exitCode=0
Nov 02 13:09:13.202523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:13.202067    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf" event={"ID":"056e1508-5e60-4712-8765-e871d3339852","Type":"ContainerDied","Data":"ebe5152b5dc9fa561079258f749e376aec7a02c81b8bee8e9df63895b4c7a139"}
Nov 02 13:09:13.390469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:13.390434    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:13.470044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:13.470003    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d2cd78d-8135-471c-b1de-176466518666-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2cd78d-8135-471c-b1de-176466518666-maas-api-openshift-ca\") pod \"5d2cd78d-8135-471c-b1de-176466518666\" (UID: \"5d2cd78d-8135-471c-b1de-176466518666\") "
Nov 02 13:09:13.470254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:13.470094    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d2cd78d-8135-471c-b1de-176466518666-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2cd78d-8135-471c-b1de-176466518666-maas-api-xks-ca\") pod \"5d2cd78d-8135-471c-b1de-176466518666\" (UID: \"5d2cd78d-8135-471c-b1de-176466518666\") "
Nov 02 13:09:13.470254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:13.470116    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2cd78d-8135-471c-b1de-176466518666-kube-api-access-nlpf4\" (UniqueName: \"kubernetes.io/projected/5d2cd78d-8135-471c-b1de-176466518666-kube-api-access-nlpf4\") pod \"5d2cd78d-8135-471c-b1de-176466518666\" (UID: \"5d2cd78d-8135-471c-b1de-176466518666\") "
Nov 02 13:09:13.470451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:13.470419    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2cd78d-8135-471c-b1de-176466518666-maas-api-openshift-ca" pod "5d2cd78d-8135-471c-b1de-176466518666" (UID: "5d2cd78d-8135-471c-b1de-176466518666"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:13.470939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:13.470761    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2cd78d-8135-471c-b1de-176466518666-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:13.472648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:13.472616    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2cd78d-8135-471c-b1de-176466518666-maas-api-xks-ca" pod "5d2cd78d-8135-471c-b1de-176466518666" (UID: "5d2cd78d-8135-471c-b1de-176466518666"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:09:13.472648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:13.472632    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2cd78d-8135-471c-b1de-176466518666-kube-api-access-nlpf4" pod "5d2cd78d-8135-471c-b1de-176466518666" (UID: "5d2cd78d-8135-471c-b1de-176466518666"). InnerVolumeSpecName "kube-api-access-nlpf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:13.572604 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:13.572520    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2cd78d-8135-471c-b1de-176466518666-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:13.572604 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:13.572554    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlpf4\" (UniqueName: \"kubernetes.io/projected/5d2cd78d-8135-471c-b1de-176466518666-kube-api-access-nlpf4\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:14.162401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.162339    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:14.212424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.212389    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"
Nov 02 13:09:14.212923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.212426    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk" event={"ID":"5d2cd78d-8135-471c-b1de-176466518666","Type":"ContainerDied","Data":"57b62d0b257890b364cdcdf95c917173f96c3ac84a9049c436f295f17edd37cc"}
Nov 02 13:09:14.212923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.212476    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57b62d0b257890b364cdcdf95c917173f96c3ac84a9049c436f295f17edd37cc"
Nov 02 13:09:14.302757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.302702    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:14.406690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.406657    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:14.485778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.485733    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/056e1508-5e60-4712-8765-e871d3339852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056e1508-5e60-4712-8765-e871d3339852-maas-api-xks-ca\") pod \"056e1508-5e60-4712-8765-e871d3339852\" (UID: \"056e1508-5e60-4712-8765-e871d3339852\") "
Nov 02 13:09:14.485985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.485813    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056e1508-5e60-4712-8765-e871d3339852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056e1508-5e60-4712-8765-e871d3339852-maas-api-openshift-ca\") pod \"056e1508-5e60-4712-8765-e871d3339852\" (UID: \"056e1508-5e60-4712-8765-e871d3339852\") "
Nov 02 13:09:14.486095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.486035    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056e1508-5e60-4712-8765-e871d3339852-kube-api-access-qs7pk\" (UniqueName: \"kubernetes.io/projected/056e1508-5e60-4712-8765-e871d3339852-kube-api-access-qs7pk\") pod \"056e1508-5e60-4712-8765-e871d3339852\" (UID: \"056e1508-5e60-4712-8765-e871d3339852\") "
Nov 02 13:09:14.486228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.486201    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056e1508-5e60-4712-8765-e871d3339852-maas-api-openshift-ca" pod "056e1508-5e60-4712-8765-e871d3339852" (UID: "056e1508-5e60-4712-8765-e871d3339852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:14.486601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.486579    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056e1508-5e60-4712-8765-e871d3339852-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:14.488799 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.488763    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056e1508-5e60-4712-8765-e871d3339852-maas-api-xks-ca" pod "056e1508-5e60-4712-8765-e871d3339852" (UID: "056e1508-5e60-4712-8765-e871d3339852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:09:14.488935 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.488893    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056e1508-5e60-4712-8765-e871d3339852-kube-api-access-qs7pk" pod "056e1508-5e60-4712-8765-e871d3339852" (UID: "056e1508-5e60-4712-8765-e871d3339852"). InnerVolumeSpecName "kube-api-access-qs7pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:14.499861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.499824    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"]
Nov 02 13:09:14.503248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.503229    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:14.524419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.524391    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"]
Nov 02 13:09:14.525210 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.525185    5879 cpu_manager.go:414] "containerMap: removing container" podUID="056e1508-5e60-4712-8765-e871d3339852" containerName="revoke-keys"
Nov 02 13:09:14.525210 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.525206    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056e1508-5e60-4712-8765-e871d3339852" containerName="revoke-keys"
Nov 02 13:09:14.525379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.525255    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5d2cd78d-8135-471c-b1de-176466518666" containerName="revoke-keys"
Nov 02 13:09:14.525379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.525261    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2cd78d-8135-471c-b1de-176466518666" containerName="revoke-keys"
Nov 02 13:09:14.525440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.525387    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2cd78d-8135-471c-b1de-176466518666" containerName="revoke-keys"
Nov 02 13:09:14.525440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.525402    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="056e1508-5e60-4712-8765-e871d3339852" containerName="revoke-keys"
Nov 02 13:09:14.529312 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.529293    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.535096 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.535070    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:14.549828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.549765    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"]
Nov 02 13:09:14.588478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.588443    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056e1508-5e60-4712-8765-e871d3339852-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:14.588478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.588471    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs7pk\" (UniqueName: \"kubernetes.io/projected/056e1508-5e60-4712-8765-e871d3339852-kube-api-access-qs7pk\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:14.693047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.692636    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-routing-overlay-volume\") pod \"payload-processing-external-model-7bb658f5fc-rjbpj\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.693047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.692772    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9c2l\" (UniqueName: \"kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-kube-api-access-v9c2l\") pod \"payload-processing-external-model-7bb658f5fc-rjbpj\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.693047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.692828    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-plugins-config-volume\") pod \"payload-processing-external-model-7bb658f5fc-rjbpj\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.693359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.693305    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-provider-credentials\") pod \"payload-processing-external-model-7bb658f5fc-rjbpj\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.794640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.794598    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-routing-overlay-volume\") pod \"payload-processing-external-model-7bb658f5fc-rjbpj\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.794866 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.794671    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9c2l\" (UniqueName: \"kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-kube-api-access-v9c2l\") pod \"payload-processing-external-model-7bb658f5fc-rjbpj\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.794866 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.794712    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-plugins-config-volume\") pod \"payload-processing-external-model-7bb658f5fc-rjbpj\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.794866 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.794829    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-provider-credentials\") pod \"payload-processing-external-model-7bb658f5fc-rjbpj\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.795521 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.795496    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-routing-overlay-volume\") pod \"payload-processing-external-model-7bb658f5fc-rjbpj\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.795950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.795912    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-plugins-config-volume\") pod \"payload-processing-external-model-7bb658f5fc-rjbpj\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.797644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.797622    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-provider-credentials\") pod \"payload-processing-external-model-7bb658f5fc-rjbpj\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.819956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.819886    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9c2l\" (UniqueName: \"kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-kube-api-access-v9c2l\") pod \"payload-processing-external-model-7bb658f5fc-rjbpj\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:14.828412 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:14.828379    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:09:14.844525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:14.844493    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:15.123202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:15.123175    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:09:15.153148 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:15.153117    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc738bb_43e3_4f63_9561_5ad94216b10d.slice/crio-ff881c2c7d817567b251d266b8e0d2f4e4f7b2e1770915561a677d9c18a877d3 WatchSource:0}: Error finding container ff881c2c7d817567b251d266b8e0d2f4e4f7b2e1770915561a677d9c18a877d3: Status 404 returned error can't find the container with id ff881c2c7d817567b251d266b8e0d2f4e4f7b2e1770915561a677d9c18a877d3
Nov 02 13:09:15.154633 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:15.154610    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"]
Nov 02 13:09:15.205677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:15.205644    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"]
Nov 02 13:09:15.226455 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:15.226433    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"
Nov 02 13:09:15.226899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:15.226877    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf" event={"ID":"056e1508-5e60-4712-8765-e871d3339852","Type":"ContainerDied","Data":"aee3420c7c39d0dedf4457aa59528022d97948b9081d37108438edf14ee70bef"}
Nov 02 13:09:15.227095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:15.226929    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aee3420c7c39d0dedf4457aa59528022d97948b9081d37108438edf14ee70bef"
Nov 02 13:09:15.228903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:15.228878    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj" event={"ID":"abc738bb-43e3-4f63-9561-5ad94216b10d","Type":"ContainerStarted","Data":"ff881c2c7d817567b251d266b8e0d2f4e4f7b2e1770915561a677d9c18a877d3"}
Nov 02 13:09:16.239151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:16.239118    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj" event={"ID":"abc738bb-43e3-4f63-9561-5ad94216b10d","Type":"ContainerStarted","Data":"b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354"}
Nov 02 13:09:16.239514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:16.239176    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:16.284114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:16.284062    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj" podStartSLOduration=2.284044785 podStartE2EDuration="2.284044785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:16.282240636 +0000 UTC m=+2502.388943488" watchObservedRunningTime="2026-11-02 13:09:16.284044785 +0000 UTC m=+2502.390747636"
Nov 02 13:09:16.364769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:16.364741    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:16.496598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:16.496500    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:17.830729 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:17.830693    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"]
Nov 02 13:09:17.862484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:17.862453    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"]
Nov 02 13:09:17.867526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:17.867502    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:17.890041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:17.888518    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-433df49ba6f6\""
Nov 02 13:09:17.890041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:17.888952    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"]
Nov 02 13:09:18.052667 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.052632    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-routing-overlay-volume\") pod \"payload-processing-external-model-6b4d54cfdc-cph7t\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") " pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.052864 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.052753    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-plugins-config-volume\") pod \"payload-processing-external-model-6b4d54cfdc-cph7t\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") " pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.052864 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.052810    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials\") pod \"payload-processing-external-model-6b4d54cfdc-cph7t\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") " pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.052937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.052877    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvng\" (UniqueName: \"kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-kube-api-access-7zvng\") pod \"payload-processing-external-model-6b4d54cfdc-cph7t\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") " pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.154113 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.154075    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-routing-overlay-volume\") pod \"payload-processing-external-model-6b4d54cfdc-cph7t\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") " pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.154306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.154160    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-plugins-config-volume\") pod \"payload-processing-external-model-6b4d54cfdc-cph7t\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") " pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.154306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.154201    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials\") pod \"payload-processing-external-model-6b4d54cfdc-cph7t\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") " pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.154306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.154260    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvng\" (UniqueName: \"kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-kube-api-access-7zvng\") pod \"payload-processing-external-model-6b4d54cfdc-cph7t\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") " pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.154795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.154764    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-routing-overlay-volume\") pod \"payload-processing-external-model-6b4d54cfdc-cph7t\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") " pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.154921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.154870    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-plugins-config-volume\") pod \"payload-processing-external-model-6b4d54cfdc-cph7t\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") " pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.157060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.157037    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials\") pod \"payload-processing-external-model-6b4d54cfdc-cph7t\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") " pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.183755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.183719    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvng\" (UniqueName: \"kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-kube-api-access-7zvng\") pod \"payload-processing-external-model-6b4d54cfdc-cph7t\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") " pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.255779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.255748    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:18.483218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.483130    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:18.791622 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:18.791590    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1d0069_ae90_4f41_94f2_739840f6a71d.slice/crio-749a007ed7eae6ecfee49204ef264c48ee387b731c65f4c23ddce0384d3f2b79 WatchSource:0}: Error finding container 749a007ed7eae6ecfee49204ef264c48ee387b731c65f4c23ddce0384d3f2b79: Status 404 returned error can't find the container with id 749a007ed7eae6ecfee49204ef264c48ee387b731c65f4c23ddce0384d3f2b79
Nov 02 13:09:18.792447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:18.792417    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"]
Nov 02 13:09:19.264225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.264185    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t" event={"ID":"ab1d0069-ae90-4f41-94f2-739840f6a71d","Type":"ContainerStarted","Data":"3bd7faad6a045f85cf952b9056d5cb624bbc0c76dcd9d175dd973013b9f8768f"}
Nov 02 13:09:19.264691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.264256    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t" event={"ID":"ab1d0069-ae90-4f41-94f2-739840f6a71d","Type":"ContainerStarted","Data":"749a007ed7eae6ecfee49204ef264c48ee387b731c65f4c23ddce0384d3f2b79"}
Nov 02 13:09:19.264691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.264292    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:19.312674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.312619    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t" podStartSLOduration=2.312602199 podStartE2EDuration="2.312602199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:19.309619564 +0000 UTC m=+2505.416322414" watchObservedRunningTime="2026-11-02 13:09:19.312602199 +0000 UTC m=+2505.419305050"
Nov 02 13:09:19.500404 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.500364    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64" podUID="3679e0cc-4500-409a-860c-2dee524e4187" containerName="payload-processing" containerID="cri-o://f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633" gracePeriod=25
Nov 02 13:09:19.783159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.783082    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:19.877506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.877455    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-kube-api-access-92tb2\" (UniqueName: \"kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-kube-api-access-92tb2\") pod \"3679e0cc-4500-409a-860c-2dee524e4187\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") "
Nov 02 13:09:19.877745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.877724    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-plugins-config-volume\") pod \"3679e0cc-4500-409a-860c-2dee524e4187\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") "
Nov 02 13:09:19.877833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.877792    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-provider-credentials\") pod \"3679e0cc-4500-409a-860c-2dee524e4187\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") "
Nov 02 13:09:19.877909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.877873    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-routing-overlay-volume\") pod \"3679e0cc-4500-409a-860c-2dee524e4187\" (UID: \"3679e0cc-4500-409a-860c-2dee524e4187\") "
Nov 02 13:09:19.878262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.878233    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-plugins-config-volume" pod "3679e0cc-4500-409a-860c-2dee524e4187" (UID: "3679e0cc-4500-409a-860c-2dee524e4187"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:19.878397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.878344    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-routing-overlay-volume" pod "3679e0cc-4500-409a-860c-2dee524e4187" (UID: "3679e0cc-4500-409a-860c-2dee524e4187"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:19.878640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.878619    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:19.878683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.878651    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3679e0cc-4500-409a-860c-2dee524e4187-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:19.880980 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.880946    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-provider-credentials" pod "3679e0cc-4500-409a-860c-2dee524e4187" (UID: "3679e0cc-4500-409a-860c-2dee524e4187"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:19.881169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.881120    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-kube-api-access-92tb2" pod "3679e0cc-4500-409a-860c-2dee524e4187" (UID: "3679e0cc-4500-409a-860c-2dee524e4187"). InnerVolumeSpecName "kube-api-access-92tb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:19.979891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.979856    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:19.979891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:19.979889    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92tb2\" (UniqueName: \"kubernetes.io/projected/3679e0cc-4500-409a-860c-2dee524e4187-kube-api-access-92tb2\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:20.084415 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.084334    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:20.206685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.206644    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96" podUID="cea38918-23c7-4e73-bb02-47c268e5305a" containerName="payload-processing" containerID="cri-o://ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425" gracePeriod=25
Nov 02 13:09:20.272967 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.272936    5879 generic.go:356] "Generic (PLEG): container finished" podID="3679e0cc-4500-409a-860c-2dee524e4187" containerID="f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633" exitCode=0
Nov 02 13:09:20.273454 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.273024    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"
Nov 02 13:09:20.273454 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.273026    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64" event={"ID":"3679e0cc-4500-409a-860c-2dee524e4187","Type":"ContainerDied","Data":"f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633"}
Nov 02 13:09:20.273454 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.273093    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64" event={"ID":"3679e0cc-4500-409a-860c-2dee524e4187","Type":"ContainerDied","Data":"fa6289b9febc5593e81078d8a90aa3efaca22f4e662adf58c460c9cc44f0dcd8"}
Nov 02 13:09:20.273454 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.273131    5879 scope.go:122] "RemoveContainer" containerID="f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633"
Nov 02 13:09:20.321268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.321246    5879 scope.go:122] "RemoveContainer" containerID="f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633"
Nov 02 13:09:20.321653 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:20.321625    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633\": container with ID starting with f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633 not found: ID does not exist" containerID="f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633"
Nov 02 13:09:20.321728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.321662    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633"} err="failed to get container status \"f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633\": rpc error: code = NotFound desc = could not find container \"f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633\": container with ID starting with f472bce826c1a9ca62ddb0a170d107c3ee9a9c605a8cd316edf4b830d949b633 not found: ID does not exist"
Nov 02 13:09:20.350500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.350414    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"]
Nov 02 13:09:20.355975 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.355938    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-9588b4f4b-jbz64"]
Nov 02 13:09:20.500745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.500721    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:09:20.587804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.587775    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-kube-api-access-x42df\" (UniqueName: \"kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-kube-api-access-x42df\") pod \"cea38918-23c7-4e73-bb02-47c268e5305a\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") "
Nov 02 13:09:20.588035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.588015    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-provider-credentials\") pod \"cea38918-23c7-4e73-bb02-47c268e5305a\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") "
Nov 02 13:09:20.588109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.588097    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-plugins-config-volume\") pod \"cea38918-23c7-4e73-bb02-47c268e5305a\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") "
Nov 02 13:09:20.588231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.588210    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-routing-overlay-volume\") pod \"cea38918-23c7-4e73-bb02-47c268e5305a\" (UID: \"cea38918-23c7-4e73-bb02-47c268e5305a\") "
Nov 02 13:09:20.588590 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.588564    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-routing-overlay-volume" pod "cea38918-23c7-4e73-bb02-47c268e5305a" (UID: "cea38918-23c7-4e73-bb02-47c268e5305a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:20.588719 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.588561    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-plugins-config-volume" pod "cea38918-23c7-4e73-bb02-47c268e5305a" (UID: "cea38918-23c7-4e73-bb02-47c268e5305a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:20.588979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.588958    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:20.589070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.588987    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cea38918-23c7-4e73-bb02-47c268e5305a-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:20.591523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.591497    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-kube-api-access-x42df" pod "cea38918-23c7-4e73-bb02-47c268e5305a" (UID: "cea38918-23c7-4e73-bb02-47c268e5305a"). InnerVolumeSpecName "kube-api-access-x42df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:20.591523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.591509    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-provider-credentials" pod "cea38918-23c7-4e73-bb02-47c268e5305a" (UID: "cea38918-23c7-4e73-bb02-47c268e5305a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:20.690688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.690653    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:20.690688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:20.690683    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x42df\" (UniqueName: \"kubernetes.io/projected/cea38918-23c7-4e73-bb02-47c268e5305a-kube-api-access-x42df\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:21.133294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.133255    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:21.270941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.270900    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:21.282824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.282791    5879 generic.go:356] "Generic (PLEG): container finished" podID="cea38918-23c7-4e73-bb02-47c268e5305a" containerID="ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425" exitCode=0
Nov 02 13:09:21.282824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.282818    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96" event={"ID":"cea38918-23c7-4e73-bb02-47c268e5305a","Type":"ContainerDied","Data":"ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425"}
Nov 02 13:09:21.283350 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.282833    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"
Nov 02 13:09:21.283350 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.282872    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96" event={"ID":"cea38918-23c7-4e73-bb02-47c268e5305a","Type":"ContainerDied","Data":"925aa465016cee6f28c7be2aa77f335bbed33616878e1c06c0a1e76d4bed0fcb"}
Nov 02 13:09:21.283350 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.282894    5879 scope.go:122] "RemoveContainer" containerID="ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425"
Nov 02 13:09:21.295152 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.295133    5879 scope.go:122] "RemoveContainer" containerID="ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425"
Nov 02 13:09:21.295447 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:21.295429    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425\": container with ID starting with ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425 not found: ID does not exist" containerID="ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425"
Nov 02 13:09:21.295511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.295457    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425"} err="failed to get container status \"ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425\": rpc error: code = NotFound desc = could not find container \"ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425\": container with ID starting with ffa2178a12a3e4c52a3850c260076d201ffe29b27b4b6f1ece7dc92a72d64425 not found: ID does not exist"
Nov 02 13:09:21.343453 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.343405    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"]
Nov 02 13:09:21.356524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.356492    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9xbr96"]
Nov 02 13:09:21.523838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.523764    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"]
Nov 02 13:09:21.524532 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.524518    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3679e0cc-4500-409a-860c-2dee524e4187" containerName="payload-processing"
Nov 02 13:09:21.524592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.524534    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3679e0cc-4500-409a-860c-2dee524e4187" containerName="payload-processing"
Nov 02 13:09:21.524592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.524563    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cea38918-23c7-4e73-bb02-47c268e5305a" containerName="payload-processing"
Nov 02 13:09:21.524592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.524569    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea38918-23c7-4e73-bb02-47c268e5305a" containerName="payload-processing"
Nov 02 13:09:21.524682 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.524666    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea38918-23c7-4e73-bb02-47c268e5305a" containerName="payload-processing"
Nov 02 13:09:21.524714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.524684    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3679e0cc-4500-409a-860c-2dee524e4187" containerName="payload-processing"
Nov 02 13:09:21.529654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.529632    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.550214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.550185    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"]
Nov 02 13:09:21.615142 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.615110    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.615309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.615173    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86z7d\" (UniqueName: \"kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-kube-api-access-86z7d\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.615424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.615401    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.615470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.615439    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.716743 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.716703    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.716743 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.716747    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.717029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.716803    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.717029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.716857    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86z7d\" (UniqueName: \"kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-kube-api-access-86z7d\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.717537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.717510    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.717661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.717597    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.719662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.719636    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.742143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.742110    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86z7d\" (UniqueName: \"kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-kube-api-access-86z7d\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.844733 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.844647    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:21.872267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:21.872237    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:22.019751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.019709    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3679e0cc-4500-409a-860c-2dee524e4187" path="/var/lib/kubelet/pods/3679e0cc-4500-409a-860c-2dee524e4187/volumes"
Nov 02 13:09:22.020221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.020203    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea38918-23c7-4e73-bb02-47c268e5305a" path="/var/lib/kubelet/pods/cea38918-23c7-4e73-bb02-47c268e5305a/volumes"
Nov 02 13:09:22.166061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.163599    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"]
Nov 02 13:09:22.176093 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:22.171763    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa981db8_0751_4762_9362_1134c53fb40c.slice/crio-1863862cf9d66cadcf17d8761f87c22f328c7a3b6cbbacef33f2ceb24a16b21c WatchSource:0}: Error finding container 1863862cf9d66cadcf17d8761f87c22f328c7a3b6cbbacef33f2ceb24a16b21c: Status 404 returned error can't find the container with id 1863862cf9d66cadcf17d8761f87c22f328c7a3b6cbbacef33f2ceb24a16b21c
Nov 02 13:09:22.225026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.224961    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"]
Nov 02 13:09:22.225452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.225421    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:22.229697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.229670    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"]
Nov 02 13:09:22.299280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.299242    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r" event={"ID":"aa981db8-0751-4762-9362-1134c53fb40c","Type":"ContainerStarted","Data":"4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0"}
Nov 02 13:09:22.299875 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.299845    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:22.300011 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.299888    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r" event={"ID":"aa981db8-0751-4762-9362-1134c53fb40c","Type":"ContainerStarted","Data":"1863862cf9d66cadcf17d8761f87c22f328c7a3b6cbbacef33f2ceb24a16b21c"}
Nov 02 13:09:22.412188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.412086    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r" podStartSLOduration=1.412069637 podStartE2EDuration="1.412069637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:22.40773359 +0000 UTC m=+2508.514436441" watchObservedRunningTime="2026-11-02 13:09:22.412069637 +0000 UTC m=+2508.518772526"
Nov 02 13:09:22.443207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.443175    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:22.660523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.660482    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:22.785565 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.785425    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:22.925214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:22.925173    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:23.206489 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.206449    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:23.255892 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.255852    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj" podUID="abc738bb-43e3-4f63-9561-5ad94216b10d" containerName="payload-processing" containerID="cri-o://b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354" gracePeriod=25
Nov 02 13:09:23.287474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.287441    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"]
Nov 02 13:09:23.358138 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:23.358092    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-90cda9/agc-cred-26e198286e55: secret "agc-cred-26e198286e55" not found
Nov 02 13:09:23.358138 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:23.358127    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk: secret "agc-cred-26e198286e55" not found
Nov 02 13:09:23.358600 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:23.358190    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials podName:0e8b12d6-622a-49d1-bd56-41a22e7f4415 nodeName:}" failed. No retries permitted until 2026-11-02 13:09:23.858170828 +0000 UTC m=+2509.964873668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk" (UID: "0e8b12d6-622a-49d1-bd56-41a22e7f4415") : secret "agc-cred-26e198286e55" not found
Nov 02 13:09:23.429019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.427964    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"]
Nov 02 13:09:23.435252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.434415    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:23.471017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.460642    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:09:23.471017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.460719    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:09:23.483057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.479782    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"]
Nov 02 13:09:23.561769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.561737    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edabc533-jpv77\" (UID: \"b5dbb607-175b-4837-bb94-6fb2710b5dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:23.562003 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.561794    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edabc533-jpv77\" (UID: \"b5dbb607-175b-4837-bb94-6fb2710b5dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:23.562003 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.561906    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9499\" (UniqueName: \"kubernetes.io/projected/b5dbb607-175b-4837-bb94-6fb2710b5dea-kube-api-access-p9499\") pod \"maas-api-revoke-sub-edabc533-jpv77\" (UID: \"b5dbb607-175b-4837-bb94-6fb2710b5dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:23.605831 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.605803    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:23.663760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.663723    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edabc533-jpv77\" (UID: \"b5dbb607-175b-4837-bb94-6fb2710b5dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:23.663978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.663810    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edabc533-jpv77\" (UID: \"b5dbb607-175b-4837-bb94-6fb2710b5dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:23.663978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.663880    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9499\" (UniqueName: \"kubernetes.io/projected/b5dbb607-175b-4837-bb94-6fb2710b5dea-kube-api-access-p9499\") pod \"maas-api-revoke-sub-edabc533-jpv77\" (UID: \"b5dbb607-175b-4837-bb94-6fb2710b5dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:23.664724 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.664643    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edabc533-jpv77\" (UID: \"b5dbb607-175b-4837-bb94-6fb2710b5dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:23.666737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.666712    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edabc533-jpv77\" (UID: \"b5dbb607-175b-4837-bb94-6fb2710b5dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:23.689268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.689234    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9499\" (UniqueName: \"kubernetes.io/projected/b5dbb607-175b-4837-bb94-6fb2710b5dea-kube-api-access-p9499\") pod \"maas-api-revoke-sub-edabc533-jpv77\" (UID: \"b5dbb607-175b-4837-bb94-6fb2710b5dea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:23.741641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.741566    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"]
Nov 02 13:09:23.742403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.742386    5879 cpu_manager.go:414] "containerMap: removing container" podUID="abc738bb-43e3-4f63-9561-5ad94216b10d" containerName="payload-processing"
Nov 02 13:09:23.742451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.742405    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc738bb-43e3-4f63-9561-5ad94216b10d" containerName="payload-processing"
Nov 02 13:09:23.742603 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.742592    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc738bb-43e3-4f63-9561-5ad94216b10d" containerName="payload-processing"
Nov 02 13:09:23.756821 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.756793    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:23.764554 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.764520    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-kube-api-access-v9c2l\" (UniqueName: \"kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-kube-api-access-v9c2l\") pod \"abc738bb-43e3-4f63-9561-5ad94216b10d\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") "
Nov 02 13:09:23.764967 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.764940    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-routing-overlay-volume\") pod \"abc738bb-43e3-4f63-9561-5ad94216b10d\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") "
Nov 02 13:09:23.765125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.765033    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-plugins-config-volume\") pod \"abc738bb-43e3-4f63-9561-5ad94216b10d\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") "
Nov 02 13:09:23.765125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.765115    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-provider-credentials\") pod \"abc738bb-43e3-4f63-9561-5ad94216b10d\" (UID: \"abc738bb-43e3-4f63-9561-5ad94216b10d\") "
Nov 02 13:09:23.765961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.765932    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-routing-overlay-volume" pod "abc738bb-43e3-4f63-9561-5ad94216b10d" (UID: "abc738bb-43e3-4f63-9561-5ad94216b10d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:23.766310 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.766291    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-plugins-config-volume" pod "abc738bb-43e3-4f63-9561-5ad94216b10d" (UID: "abc738bb-43e3-4f63-9561-5ad94216b10d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:23.767634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.767612    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"]
Nov 02 13:09:23.768077 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.768058    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-kube-api-access-v9c2l" pod "abc738bb-43e3-4f63-9561-5ad94216b10d" (UID: "abc738bb-43e3-4f63-9561-5ad94216b10d"). InnerVolumeSpecName "kube-api-access-v9c2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:23.768434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.768413    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-provider-credentials" pod "abc738bb-43e3-4f63-9561-5ad94216b10d" (UID: "abc738bb-43e3-4f63-9561-5ad94216b10d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:23.836727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.822259    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:23.875626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.874592    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:23.875626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.874682    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42cm\" (UniqueName: \"kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-kube-api-access-x42cm\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:23.875626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.874725    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:23.875626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.874788    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:23.875626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.874860    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9c2l\" (UniqueName: \"kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-kube-api-access-v9c2l\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:23.875626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.874874    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:23.875626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.874886    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abc738bb-43e3-4f63-9561-5ad94216b10d-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:23.875626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.874897    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abc738bb-43e3-4f63-9561-5ad94216b10d-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:23.875626 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:23.875080    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-90cda9/agc-cred-26e198286e55: secret "agc-cred-26e198286e55" not found
Nov 02 13:09:23.875626 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:23.875101    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk: secret "agc-cred-26e198286e55" not found
Nov 02 13:09:23.875626 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:23.875156    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials podName:0e8b12d6-622a-49d1-bd56-41a22e7f4415 nodeName:}" failed. No retries permitted until 2026-11-02 13:09:24.875137491 +0000 UTC m=+2510.981840335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk" (UID: "0e8b12d6-622a-49d1-bd56-41a22e7f4415") : secret "agc-cred-26e198286e55" not found
Nov 02 13:09:23.960554 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.960509    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:23.981632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.981273    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:23.982025 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.981973    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42cm\" (UniqueName: \"kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-kube-api-access-x42cm\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:23.982234 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.982215    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:23.982411 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.982396    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:23.983271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.983200    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:23.983510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.983484    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:23.987440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:23.987414    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:24.007808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.007728    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42cm\" (UniqueName: \"kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-kube-api-access-x42cm\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:24.077182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.077147    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:24.096572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.096538    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:24.238949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.238898    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"]
Nov 02 13:09:24.348759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.338237    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77" event={"ID":"b5dbb607-175b-4837-bb94-6fb2710b5dea","Type":"ContainerStarted","Data":"e3c9a3668298d6ed50fc24e3c7b18106640f5d84bd12aac95214fd2dbd5e8d81"}
Nov 02 13:09:24.348759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.340770    5879 generic.go:356] "Generic (PLEG): container finished" podID="abc738bb-43e3-4f63-9561-5ad94216b10d" containerID="b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354" exitCode=0
Nov 02 13:09:24.348759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.340843    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"
Nov 02 13:09:24.348759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.340878    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj" event={"ID":"abc738bb-43e3-4f63-9561-5ad94216b10d","Type":"ContainerDied","Data":"b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354"}
Nov 02 13:09:24.348759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.340920    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj" event={"ID":"abc738bb-43e3-4f63-9561-5ad94216b10d","Type":"ContainerDied","Data":"ff881c2c7d817567b251d266b8e0d2f4e4f7b2e1770915561a677d9c18a877d3"}
Nov 02 13:09:24.348759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.341722    5879 scope.go:122] "RemoveContainer" containerID="b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354"
Nov 02 13:09:24.385226 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.383826    5879 scope.go:122] "RemoveContainer" containerID="b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354"
Nov 02 13:09:24.386059 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:24.385921    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354\": container with ID starting with b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354 not found: ID does not exist" containerID="b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354"
Nov 02 13:09:24.386059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.385967    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354"} err="failed to get container status \"b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354\": rpc error: code = NotFound desc = could not find container \"b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354\": container with ID starting with b64d65bae19ecab370fcbeac7fe6bd92e5271f578e67e02617be40907c8cd354 not found: ID does not exist"
Nov 02 13:09:24.417121 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:24.417082    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8c6740_cb02_4212_975b_8203ef99d355.slice/crio-9867048fa524038050e979f3430d89d81bf9fdf574295189f91eceade9a50672 WatchSource:0}: Error finding container 9867048fa524038050e979f3430d89d81bf9fdf574295189f91eceade9a50672: Status 404 returned error can't find the container with id 9867048fa524038050e979f3430d89d81bf9fdf574295189f91eceade9a50672
Nov 02 13:09:24.418106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.418019    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"]
Nov 02 13:09:24.422941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.422913    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"]
Nov 02 13:09:24.427043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:24.427017    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-rjbpj"]
Nov 02 13:09:24.936020 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:24.935975    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-90cda9/agc-cred-26e198286e55: secret "agc-cred-26e198286e55" not found
Nov 02 13:09:24.936020 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:24.936018    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk: secret "agc-cred-26e198286e55" not found
Nov 02 13:09:24.936272 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:24.936070    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials podName:0e8b12d6-622a-49d1-bd56-41a22e7f4415 nodeName:}" failed. No retries permitted until 2026-11-02 13:09:26.936056139 +0000 UTC m=+2513.042758968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk" (UID: "0e8b12d6-622a-49d1-bd56-41a22e7f4415") : secret "agc-cred-26e198286e55" not found
Nov 02 13:09:25.353761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:25.353725    5879 generic.go:356] "Generic (PLEG): container finished" podID="b5dbb607-175b-4837-bb94-6fb2710b5dea" containerID="0f273e2670dda97528850879e8273f86933cf1fed880129204cac533a18bafba" exitCode=0
Nov 02 13:09:25.353928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:25.353770    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77" event={"ID":"b5dbb607-175b-4837-bb94-6fb2710b5dea","Type":"ContainerDied","Data":"0f273e2670dda97528850879e8273f86933cf1fed880129204cac533a18bafba"}
Nov 02 13:09:25.356863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:25.356832    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57" event={"ID":"7b8c6740-cb02-4212-975b-8203ef99d355","Type":"ContainerStarted","Data":"c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4"}
Nov 02 13:09:25.356981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:25.356879    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57" event={"ID":"7b8c6740-cb02-4212-975b-8203ef99d355","Type":"ContainerStarted","Data":"9867048fa524038050e979f3430d89d81bf9fdf574295189f91eceade9a50672"}
Nov 02 13:09:25.356981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:25.356923    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:25.446304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:25.446252    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57" podStartSLOduration=2.446236777 podStartE2EDuration="2.446236777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:25.444538173 +0000 UTC m=+2511.551241025" watchObservedRunningTime="2026-11-02 13:09:25.446236777 +0000 UTC m=+2511.552939674"
Nov 02 13:09:25.869440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:25.869407    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:26.019525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:26.019478    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc738bb-43e3-4f63-9561-5ad94216b10d" path="/var/lib/kubelet/pods/abc738bb-43e3-4f63-9561-5ad94216b10d/volumes"
Nov 02 13:09:26.531192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:26.531165    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:26.566268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:26.566234    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5dbb607-175b-4837-bb94-6fb2710b5dea-kube-api-access-p9499\" (UniqueName: \"kubernetes.io/projected/b5dbb607-175b-4837-bb94-6fb2710b5dea-kube-api-access-p9499\") pod \"b5dbb607-175b-4837-bb94-6fb2710b5dea\" (UID: \"b5dbb607-175b-4837-bb94-6fb2710b5dea\") "
Nov 02 13:09:26.566425 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:26.566353    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-openshift-ca\") pod \"b5dbb607-175b-4837-bb94-6fb2710b5dea\" (UID: \"b5dbb607-175b-4837-bb94-6fb2710b5dea\") "
Nov 02 13:09:26.566425 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:26.566401    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-xks-ca\") pod \"b5dbb607-175b-4837-bb94-6fb2710b5dea\" (UID: \"b5dbb607-175b-4837-bb94-6fb2710b5dea\") "
Nov 02 13:09:26.566787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:26.566759    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-openshift-ca" pod "b5dbb607-175b-4837-bb94-6fb2710b5dea" (UID: "b5dbb607-175b-4837-bb94-6fb2710b5dea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:26.567127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:26.567109    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:26.568677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:26.568648    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-xks-ca" pod "b5dbb607-175b-4837-bb94-6fb2710b5dea" (UID: "b5dbb607-175b-4837-bb94-6fb2710b5dea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:09:26.568677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:26.568662    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5dbb607-175b-4837-bb94-6fb2710b5dea-kube-api-access-p9499" pod "b5dbb607-175b-4837-bb94-6fb2710b5dea" (UID: "b5dbb607-175b-4837-bb94-6fb2710b5dea"). InnerVolumeSpecName "kube-api-access-p9499". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:26.668724 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:26.668628    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9499\" (UniqueName: \"kubernetes.io/projected/b5dbb607-175b-4837-bb94-6fb2710b5dea-kube-api-access-p9499\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:26.668724 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:26.668658    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5dbb607-175b-4837-bb94-6fb2710b5dea-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:26.972903 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:26.972803    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-90cda9/agc-cred-26e198286e55: secret "agc-cred-26e198286e55" not found
Nov 02 13:09:26.972903 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:26.972839    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk: secret "agc-cred-26e198286e55" not found
Nov 02 13:09:26.972903 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:26.972899    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials podName:0e8b12d6-622a-49d1-bd56-41a22e7f4415 nodeName:}" failed. No retries permitted until 2026-11-02 13:09:30.972882792 +0000 UTC m=+2517.079585644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk" (UID: "0e8b12d6-622a-49d1-bd56-41a22e7f4415") : secret "agc-cred-26e198286e55" not found
Nov 02 13:09:27.162840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.162802    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"]
Nov 02 13:09:27.163671 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.163655    5879 cpu_manager.go:414] "containerMap: removing container" podUID="b5dbb607-175b-4837-bb94-6fb2710b5dea" containerName="revoke-keys"
Nov 02 13:09:27.163733 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.163673    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5dbb607-175b-4837-bb94-6fb2710b5dea" containerName="revoke-keys"
Nov 02 13:09:27.163797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.163788    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5dbb607-175b-4837-bb94-6fb2710b5dea" containerName="revoke-keys"
Nov 02 13:09:27.167977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.167952    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.210809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.210772    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"]
Nov 02 13:09:27.225402 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.225262    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt" podUID="fa28ea8c-32f7-4962-9528-dfd1c10dd44a" containerName="payload-processing" containerID="cri-o://8f350e5bf01112371bd2de757c21c243cacb8a8c43b369eec0d5cbde8ec921f8" gracePeriod=25
Nov 02 13:09:27.276436 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.276390    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.276594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.276447    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.276594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.276559    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.276699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.276680    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.276765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.276749    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.276811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.276798    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.276936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.276925    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.276981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.276969    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twj2k\" (UniqueName: \"kubernetes.io/projected/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-kube-api-access-twj2k\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.277101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.277081    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.287864 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:27.287834    5879 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa28ea8c_32f7_4962_9528_dfd1c10dd44a.slice/crio-conmon-8f350e5bf01112371bd2de757c21c243cacb8a8c43b369eec0d5cbde8ec921f8.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 13:09:27.299520 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.299467    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r" podUID="aa981db8-0751-4762-9362-1134c53fb40c" containerName="payload-processing" containerID="cri-o://4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0" gracePeriod=25
Nov 02 13:09:27.375920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.375885    5879 generic.go:356] "Generic (PLEG): container finished" podID="fa28ea8c-32f7-4962-9528-dfd1c10dd44a" containerID="8f350e5bf01112371bd2de757c21c243cacb8a8c43b369eec0d5cbde8ec921f8" exitCode=0
Nov 02 13:09:27.376129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.375960    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt" event={"ID":"fa28ea8c-32f7-4962-9528-dfd1c10dd44a","Type":"ContainerDied","Data":"8f350e5bf01112371bd2de757c21c243cacb8a8c43b369eec0d5cbde8ec921f8"}
Nov 02 13:09:27.377682 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.377655    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"
Nov 02 13:09:27.377824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.377678    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77" event={"ID":"b5dbb607-175b-4837-bb94-6fb2710b5dea","Type":"ContainerDied","Data":"e3c9a3668298d6ed50fc24e3c7b18106640f5d84bd12aac95214fd2dbd5e8d81"}
Nov 02 13:09:27.377824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.377727    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3c9a3668298d6ed50fc24e3c7b18106640f5d84bd12aac95214fd2dbd5e8d81"
Nov 02 13:09:27.378119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.378100    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.378165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.378136    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.378221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.378161    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.378595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.378572    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.378728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.378615    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.378728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.378650    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.378728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.378666    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.378882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.378800    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.378882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.378858    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twj2k\" (UniqueName: \"kubernetes.io/projected/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-kube-api-access-twj2k\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.379106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.378884    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.379106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.378918    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.379106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.378969    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.379347 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.379326    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.379537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.379518    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.382360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.382332    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.382360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.382345    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.421903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.421875    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.422302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.422282    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twj2k\" (UniqueName: \"kubernetes.io/projected/c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d-kube-api-access-twj2k\") pod \"maas-default-gateway-openshift-default-5bc884dd87-27twq\" (UID: \"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.492740 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.492409    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:27.639313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.638242    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:09:27.641896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.641870    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:27.705764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.705732    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-plugins-config-volume\") pod \"aa981db8-0751-4762-9362-1134c53fb40c\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") "
Nov 02 13:09:27.705949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.705824    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-routing-overlay-volume\") pod \"aa981db8-0751-4762-9362-1134c53fb40c\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") "
Nov 02 13:09:27.705949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.705885    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-kube-api-access-mfqsz\" (UniqueName: \"kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-kube-api-access-mfqsz\") pod \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") "
Nov 02 13:09:27.705949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.705912    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-provider-credentials\") pod \"aa981db8-0751-4762-9362-1134c53fb40c\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") "
Nov 02 13:09:27.706162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.706081    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-kube-api-access-86z7d\" (UniqueName: \"kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-kube-api-access-86z7d\") pod \"aa981db8-0751-4762-9362-1134c53fb40c\" (UID: \"aa981db8-0751-4762-9362-1134c53fb40c\") "
Nov 02 13:09:27.706162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.706131    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-routing-overlay-volume\") pod \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") "
Nov 02 13:09:27.706300 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.706194    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-provider-credentials\") pod \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") "
Nov 02 13:09:27.706364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.706289    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-plugins-config-volume" pod "aa981db8-0751-4762-9362-1134c53fb40c" (UID: "aa981db8-0751-4762-9362-1134c53fb40c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:27.706364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.706311    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-plugins-config-volume\") pod \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\" (UID: \"fa28ea8c-32f7-4962-9528-dfd1c10dd44a\") "
Nov 02 13:09:27.706364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.706312    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-routing-overlay-volume" pod "aa981db8-0751-4762-9362-1134c53fb40c" (UID: "aa981db8-0751-4762-9362-1134c53fb40c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:27.706710 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.706683    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-routing-overlay-volume" pod "fa28ea8c-32f7-4962-9528-dfd1c10dd44a" (UID: "fa28ea8c-32f7-4962-9528-dfd1c10dd44a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:27.707141 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.707067    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-plugins-config-volume" pod "fa28ea8c-32f7-4962-9528-dfd1c10dd44a" (UID: "fa28ea8c-32f7-4962-9528-dfd1c10dd44a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:27.707283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.707170    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:27.707283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.707194    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:27.707411 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.707323    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aa981db8-0751-4762-9362-1134c53fb40c-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:27.709053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.709026    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-kube-api-access-86z7d" pod "aa981db8-0751-4762-9362-1134c53fb40c" (UID: "aa981db8-0751-4762-9362-1134c53fb40c"). InnerVolumeSpecName "kube-api-access-86z7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:27.709167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.709075    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-provider-credentials" pod "aa981db8-0751-4762-9362-1134c53fb40c" (UID: "aa981db8-0751-4762-9362-1134c53fb40c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:27.709286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.709270    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-provider-credentials" pod "fa28ea8c-32f7-4962-9528-dfd1c10dd44a" (UID: "fa28ea8c-32f7-4962-9528-dfd1c10dd44a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:27.709463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.709447    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-kube-api-access-mfqsz" pod "fa28ea8c-32f7-4962-9528-dfd1c10dd44a" (UID: "fa28ea8c-32f7-4962-9528-dfd1c10dd44a"). InnerVolumeSpecName "kube-api-access-mfqsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:27.809459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.809364    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:27.809459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.809402    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfqsz\" (UniqueName: \"kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-kube-api-access-mfqsz\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:27.809459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.809417    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:27.809459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.809431    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86z7d\" (UniqueName: \"kubernetes.io/projected/aa981db8-0751-4762-9362-1134c53fb40c-kube-api-access-86z7d\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:27.809459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.809445    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fa28ea8c-32f7-4962-9528-dfd1c10dd44a-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:27.857164 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:27.857122    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc01e2f4e_f5ca_4231_bc14_be2fa6b4b59d.slice/crio-5f219bb8f90b991811dc6596c584d4970214d3303acbdd44448635e9c1748a49 WatchSource:0}: Error finding container 5f219bb8f90b991811dc6596c584d4970214d3303acbdd44448635e9c1748a49: Status 404 returned error can't find the container with id 5f219bb8f90b991811dc6596c584d4970214d3303acbdd44448635e9c1748a49
Nov 02 13:09:27.858739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.858715    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:27.859244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.859221    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"]
Nov 02 13:09:27.860584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.860552    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:09:27.860659 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.860633    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:09:27.860701 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:27.860663    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:09:28.288273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.288234    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h" podUID="e5ba9b93-367e-4b2a-83c9-29cddfdda3ac" containerName="payload-processing" containerID="cri-o://07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063" gracePeriod=25
Nov 02 13:09:28.390441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.390403    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" event={"ID":"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d","Type":"ContainerStarted","Data":"3c8279b78ec056617f5b3aa730c3c0086fd5ed6673bdbfe258c9fef29c13a13e"}
Nov 02 13:09:28.390666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.390473    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" event={"ID":"c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d","Type":"ContainerStarted","Data":"5f219bb8f90b991811dc6596c584d4970214d3303acbdd44448635e9c1748a49"}
Nov 02 13:09:28.393497 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.393473    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"
Nov 02 13:09:28.393640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.393471    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt" event={"ID":"fa28ea8c-32f7-4962-9528-dfd1c10dd44a","Type":"ContainerDied","Data":"64f60c732c771370dfaa074d88c25ca5acad3acee5200f93a0d764c3b3990f51"}
Nov 02 13:09:28.393640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.393625    5879 scope.go:122] "RemoveContainer" containerID="8f350e5bf01112371bd2de757c21c243cacb8a8c43b369eec0d5cbde8ec921f8"
Nov 02 13:09:28.395529 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.395500    5879 generic.go:356] "Generic (PLEG): container finished" podID="aa981db8-0751-4762-9362-1134c53fb40c" containerID="4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0" exitCode=0
Nov 02 13:09:28.395710 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.395606    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"
Nov 02 13:09:28.396178 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.395623    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r" event={"ID":"aa981db8-0751-4762-9362-1134c53fb40c","Type":"ContainerDied","Data":"4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0"}
Nov 02 13:09:28.396361 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.396215    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r" event={"ID":"aa981db8-0751-4762-9362-1134c53fb40c","Type":"ContainerDied","Data":"1863862cf9d66cadcf17d8761f87c22f328c7a3b6cbbacef33f2ceb24a16b21c"}
Nov 02 13:09:28.405635 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.405615    5879 scope.go:122] "RemoveContainer" containerID="4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0"
Nov 02 13:09:28.417158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.417136    5879 scope.go:122] "RemoveContainer" containerID="4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0"
Nov 02 13:09:28.417558 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:28.417534    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0\": container with ID starting with 4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0 not found: ID does not exist" containerID="4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0"
Nov 02 13:09:28.417652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.417572    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0"} err="failed to get container status \"4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0\": rpc error: code = NotFound desc = could not find container \"4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0\": container with ID starting with 4b1596cffb477753abb6c1765d7bc92f804d71aa18812727f39a15a62f535ae0 not found: ID does not exist"
Nov 02 13:09:28.440153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.440098    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podStartSLOduration=1.4400844099999999 podStartE2EDuration="1.44008441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:28.438885723 +0000 UTC m=+2514.545588574" watchObservedRunningTime="2026-11-02 13:09:28.44008441 +0000 UTC m=+2514.546787261"
Nov 02 13:09:28.489305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.485930    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"]
Nov 02 13:09:28.490822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.490792    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-d8fck5rgt"]
Nov 02 13:09:28.493626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.493604    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:28.495254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.495225    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-27twq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:09:28.495421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.495286    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podUID="c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 02 13:09:28.527056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.527028    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"]
Nov 02 13:09:28.533647 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.533607    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64ccxj62r"]
Nov 02 13:09:28.579265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.579237    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:09:28.725638 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.725586    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-plugins-config-volume\") pod \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") "
Nov 02 13:09:28.726228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.725668    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-routing-overlay-volume\") pod \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") "
Nov 02 13:09:28.726228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.725764    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-provider-credentials\") pod \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") "
Nov 02 13:09:28.726228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.725808    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-kube-api-access-rczdm\" (UniqueName: \"kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-kube-api-access-rczdm\") pod \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\" (UID: \"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac\") "
Nov 02 13:09:28.726228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.726123    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-routing-overlay-volume" pod "e5ba9b93-367e-4b2a-83c9-29cddfdda3ac" (UID: "e5ba9b93-367e-4b2a-83c9-29cddfdda3ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:28.726454 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.726293    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-plugins-config-volume" pod "e5ba9b93-367e-4b2a-83c9-29cddfdda3ac" (UID: "e5ba9b93-367e-4b2a-83c9-29cddfdda3ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:28.726737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.726710    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:28.726802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.726743    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:28.728401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.728376    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-kube-api-access-rczdm" pod "e5ba9b93-367e-4b2a-83c9-29cddfdda3ac" (UID: "e5ba9b93-367e-4b2a-83c9-29cddfdda3ac"). InnerVolumeSpecName "kube-api-access-rczdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:28.728544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.728532    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-provider-credentials" pod "e5ba9b93-367e-4b2a-83c9-29cddfdda3ac" (UID: "e5ba9b93-367e-4b2a-83c9-29cddfdda3ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:28.828492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.828404    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:28.828492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:28.828436    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rczdm\" (UniqueName: \"kubernetes.io/projected/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac-kube-api-access-rczdm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:29.421220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.421188    5879 generic.go:356] "Generic (PLEG): container finished" podID="e5ba9b93-367e-4b2a-83c9-29cddfdda3ac" containerID="07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063" exitCode=0
Nov 02 13:09:29.421399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.421250    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"
Nov 02 13:09:29.421399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.421265    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h" event={"ID":"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac","Type":"ContainerDied","Data":"07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063"}
Nov 02 13:09:29.421399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.421319    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h" event={"ID":"e5ba9b93-367e-4b2a-83c9-29cddfdda3ac","Type":"ContainerDied","Data":"f602c80d3420ac7ac78d190faf60684406d81572d435c2874764d1c37671f7b0"}
Nov 02 13:09:29.421399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.421350    5879 scope.go:122] "RemoveContainer" containerID="07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063"
Nov 02 13:09:29.434032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.434007    5879 scope.go:122] "RemoveContainer" containerID="07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063"
Nov 02 13:09:29.434500 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:29.434468    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063\": container with ID starting with 07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063 not found: ID does not exist" containerID="07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063"
Nov 02 13:09:29.434603 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.434499    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063"} err="failed to get container status \"07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063\": rpc error: code = NotFound desc = could not find container \"07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063\": container with ID starting with 07ac89b33ab24c435047b2c2e1850a705877867c9073788e1fcd799967679063 not found: ID does not exist"
Nov 02 13:09:29.463865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.463833    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:29.479078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.479047    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"]
Nov 02 13:09:29.512684 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.512648    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-27twq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:09:29.512873 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.512723    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podUID="c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 02 13:09:29.515067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.515042    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-69bc9dr5h"]
Nov 02 13:09:29.608176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.608143    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:29.748766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.748684    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:29.872113 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:29.872082    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:30.026881 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.026784    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa981db8-0751-4762-9362-1134c53fb40c" path="/var/lib/kubelet/pods/aa981db8-0751-4762-9362-1134c53fb40c/volumes"
Nov 02 13:09:30.027405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.027381    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ba9b93-367e-4b2a-83c9-29cddfdda3ac" path="/var/lib/kubelet/pods/e5ba9b93-367e-4b2a-83c9-29cddfdda3ac/volumes"
Nov 02 13:09:30.027966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.027944    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa28ea8c-32f7-4962-9528-dfd1c10dd44a" path="/var/lib/kubelet/pods/fa28ea8c-32f7-4962-9528-dfd1c10dd44a/volumes"
Nov 02 13:09:30.166358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.166327    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:30.278075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.277957    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:30.297698 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.297662    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:30.354096 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:30.353705    5879 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-433df49ba6f6: secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:30.354096 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:30.353735    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t: secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:30.354096 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:30.353791    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials podName:ab1d0069-ae90-4f41-94f2-739840f6a71d nodeName:}" failed. No retries permitted until 2026-11-02 13:09:30.853772302 +0000 UTC m=+2516.960475137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials") pod "payload-processing-external-model-6b4d54cfdc-cph7t" (UID: "ab1d0069-ae90-4f41-94f2-739840f6a71d") : secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:30.365589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.364607    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"]
Nov 02 13:09:30.505031 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.504493    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-27twq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:09:30.505031 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.504555    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podUID="c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 02 13:09:30.627046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.627009    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"]
Nov 02 13:09:30.627795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.627771    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e5ba9b93-367e-4b2a-83c9-29cddfdda3ac" containerName="payload-processing"
Nov 02 13:09:30.627795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.627788    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ba9b93-367e-4b2a-83c9-29cddfdda3ac" containerName="payload-processing"
Nov 02 13:09:30.627795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.627801    5879 cpu_manager.go:414] "containerMap: removing container" podUID="fa28ea8c-32f7-4962-9528-dfd1c10dd44a" containerName="payload-processing"
Nov 02 13:09:30.628060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.627807    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa28ea8c-32f7-4962-9528-dfd1c10dd44a" containerName="payload-processing"
Nov 02 13:09:30.628060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.627814    5879 cpu_manager.go:414] "containerMap: removing container" podUID="aa981db8-0751-4762-9362-1134c53fb40c" containerName="payload-processing"
Nov 02 13:09:30.628060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.627819    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa981db8-0751-4762-9362-1134c53fb40c" containerName="payload-processing"
Nov 02 13:09:30.628060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.627953    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ba9b93-367e-4b2a-83c9-29cddfdda3ac" containerName="payload-processing"
Nov 02 13:09:30.628060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.627961    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa28ea8c-32f7-4962-9528-dfd1c10dd44a" containerName="payload-processing"
Nov 02 13:09:30.628060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.627969    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa981db8-0751-4762-9362-1134c53fb40c" containerName="payload-processing"
Nov 02 13:09:30.633562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.633536    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.652408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.652375    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"]
Nov 02 13:09:30.758809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.758766    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-plugins-config-volume\") pod \"payload-processing-external-model-7bb658f5fc-c8nzt\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.759277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.758837    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmj7\" (UniqueName: \"kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-kube-api-access-vvmj7\") pod \"payload-processing-external-model-7bb658f5fc-c8nzt\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.759277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.758940    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-routing-overlay-volume\") pod \"payload-processing-external-model-7bb658f5fc-c8nzt\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.759277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.758980    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-provider-credentials\") pod \"payload-processing-external-model-7bb658f5fc-c8nzt\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.861012 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.860964    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-plugins-config-volume\") pod \"payload-processing-external-model-7bb658f5fc-c8nzt\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.861184 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:30.861125    5879 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-433df49ba6f6: secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:30.861184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.861139    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmj7\" (UniqueName: \"kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-kube-api-access-vvmj7\") pod \"payload-processing-external-model-7bb658f5fc-c8nzt\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.861317 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:30.861148    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t: secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:30.861317 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:30.861280    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials podName:ab1d0069-ae90-4f41-94f2-739840f6a71d nodeName:}" failed. No retries permitted until 2026-11-02 13:09:31.86126159 +0000 UTC m=+2517.967964433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials") pod "payload-processing-external-model-6b4d54cfdc-cph7t" (UID: "ab1d0069-ae90-4f41-94f2-739840f6a71d") : secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:30.861457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.861215    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-routing-overlay-volume\") pod \"payload-processing-external-model-7bb658f5fc-c8nzt\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.861457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.861394    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-provider-credentials\") pod \"payload-processing-external-model-7bb658f5fc-c8nzt\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.861768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.861750    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-routing-overlay-volume\") pod \"payload-processing-external-model-7bb658f5fc-c8nzt\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.861865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.861846    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-plugins-config-volume\") pod \"payload-processing-external-model-7bb658f5fc-c8nzt\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.864152 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.864127    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-provider-credentials\") pod \"payload-processing-external-model-7bb658f5fc-c8nzt\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.908113 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.908024    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmj7\" (UniqueName: \"kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-kube-api-access-vvmj7\") pod \"payload-processing-external-model-7bb658f5fc-c8nzt\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:30.955634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:30.955269    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:31.067800 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:31.067616    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-90cda9/agc-cred-26e198286e55: secret "agc-cred-26e198286e55" not found
Nov 02 13:09:31.067800 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:31.067694    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk: secret "agc-cred-26e198286e55" not found
Nov 02 13:09:31.067800 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:31.067762    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials podName:0e8b12d6-622a-49d1-bd56-41a22e7f4415 nodeName:}" failed. No retries permitted until 2026-11-02 13:09:39.067739225 +0000 UTC m=+2525.174442077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk" (UID: "0e8b12d6-622a-49d1-bd56-41a22e7f4415") : secret "agc-cred-26e198286e55" not found
Nov 02 13:09:31.317123 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:31.317082    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea67639_2f9f_41c7_bdc2_ab599a384285.slice/crio-0fd482c904b68625a89725d93d26f0f4032f59521a428ced197bbbd60f50bbf6 WatchSource:0}: Error finding container 0fd482c904b68625a89725d93d26f0f4032f59521a428ced197bbbd60f50bbf6: Status 404 returned error can't find the container with id 0fd482c904b68625a89725d93d26f0f4032f59521a428ced197bbbd60f50bbf6
Nov 02 13:09:31.318287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:31.318261    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"]
Nov 02 13:09:31.443815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:31.443785    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt" event={"ID":"5ea67639-2f9f-41c7-bdc2-ab599a384285","Type":"ContainerStarted","Data":"c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea"}
Nov 02 13:09:31.444017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:31.443838    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt" event={"ID":"5ea67639-2f9f-41c7-bdc2-ab599a384285","Type":"ContainerStarted","Data":"0fd482c904b68625a89725d93d26f0f4032f59521a428ced197bbbd60f50bbf6"}
Nov 02 13:09:31.444017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:31.443861    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:31.501541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:31.501502    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-27twq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:09:31.501841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:31.501809    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podUID="c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 02 13:09:31.512676 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:31.512617    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt" podStartSLOduration=1.5126014049999998 podStartE2EDuration="1.512601405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:31.502420861 +0000 UTC m=+2517.609123713" watchObservedRunningTime="2026-11-02 13:09:31.512601405 +0000 UTC m=+2517.619304288"
Nov 02 13:09:31.791815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:31.791685    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:31.878685 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:31.878657    5879 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-433df49ba6f6: secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:31.878685 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:31.878686    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t: secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:31.878893 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:31.878738    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials podName:ab1d0069-ae90-4f41-94f2-739840f6a71d nodeName:}" failed. No retries permitted until 2026-11-02 13:09:33.878723723 +0000 UTC m=+2519.985426552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials") pod "payload-processing-external-model-6b4d54cfdc-cph7t" (UID: "ab1d0069-ae90-4f41-94f2-739840f6a71d") : secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:32.055839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:32.055656    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:32.176447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:32.176397    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:32.498016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:32.494532    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-27twq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:09:32.498016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:32.494589    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podUID="c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 02 13:09:33.494019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:33.493441    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-27twq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:09:33.494019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:33.493504    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podUID="c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 02 13:09:33.761728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:33.761628    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"]
Nov 02 13:09:33.810582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:33.810526    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"]
Nov 02 13:09:33.815963 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:33.815930    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:33.822145 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:33.822118    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"]
Nov 02 13:09:33.833953 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:33.833923    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7f082accea24\""
Nov 02 13:09:33.870457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:33.870430    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:33.937757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:33.937721    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-routing-overlay-volume\") pod \"payload-processing-external-model-666978d5bb-cjbj2\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") " pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:33.938095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:33.938073    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwmd7\" (UniqueName: \"kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-kube-api-access-kwmd7\") pod \"payload-processing-external-model-666978d5bb-cjbj2\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") " pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:33.938401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:33.938384    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-plugins-config-volume\") pod \"payload-processing-external-model-666978d5bb-cjbj2\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") " pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:33.938721 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:33.938703    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-provider-credentials\") pod \"payload-processing-external-model-666978d5bb-cjbj2\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") " pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:33.939130 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:33.939112    5879 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-433df49ba6f6: secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:33.939267 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:33.939256    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t: secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:33.939413 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:33.939403    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials podName:ab1d0069-ae90-4f41-94f2-739840f6a71d nodeName:}" failed. No retries permitted until 2026-11-02 13:09:37.939382615 +0000 UTC m=+2524.046085464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials") pod "payload-processing-external-model-6b4d54cfdc-cph7t" (UID: "ab1d0069-ae90-4f41-94f2-739840f6a71d") : secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:34.043925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.043837    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-provider-credentials\") pod \"payload-processing-external-model-666978d5bb-cjbj2\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") " pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:34.050713 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.050629    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-routing-overlay-volume\") pod \"payload-processing-external-model-666978d5bb-cjbj2\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") " pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:34.050893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.050715    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwmd7\" (UniqueName: \"kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-kube-api-access-kwmd7\") pod \"payload-processing-external-model-666978d5bb-cjbj2\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") " pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:34.050893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.050821    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-plugins-config-volume\") pod \"payload-processing-external-model-666978d5bb-cjbj2\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") " pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:34.052615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.052557    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-routing-overlay-volume\") pod \"payload-processing-external-model-666978d5bb-cjbj2\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") " pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:34.052615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.052607    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-plugins-config-volume\") pod \"payload-processing-external-model-666978d5bb-cjbj2\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") " pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:34.065273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.065247    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7f082accea24\""
Nov 02 13:09:34.073705 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.073674    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-provider-credentials\") pod \"payload-processing-external-model-666978d5bb-cjbj2\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") " pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:34.077590 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.077563    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwmd7\" (UniqueName: \"kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-kube-api-access-kwmd7\") pod \"payload-processing-external-model-666978d5bb-cjbj2\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") " pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:34.140295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.140255    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:34.493804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.493765    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-27twq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:09:34.494416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.493826    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podUID="c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 02 13:09:34.516823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:34.516796    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"]
Nov 02 13:09:35.095814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.095765    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:35.236612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.236261    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:35.365559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.365508    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55" podUID="f4217d62-9037-41fd-9279-5c59120431a3" containerName="payload-processing" containerID="cri-o://8d741c4a9e9d9969505687746a87cfc174bfbf5a77077ff52046e544f7dbf9ce" gracePeriod=25
Nov 02 13:09:35.486935 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.486905    5879 generic.go:356] "Generic (PLEG): container finished" podID="f4217d62-9037-41fd-9279-5c59120431a3" containerID="8d741c4a9e9d9969505687746a87cfc174bfbf5a77077ff52046e544f7dbf9ce" exitCode=0
Nov 02 13:09:35.487230 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.487120    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55" event={"ID":"f4217d62-9037-41fd-9279-5c59120431a3","Type":"ContainerDied","Data":"8d741c4a9e9d9969505687746a87cfc174bfbf5a77077ff52046e544f7dbf9ce"}
Nov 02 13:09:35.489628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.489593    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2" event={"ID":"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d","Type":"ContainerStarted","Data":"01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691"}
Nov 02 13:09:35.489772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.489655    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2" event={"ID":"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d","Type":"ContainerStarted","Data":"5766a96011aeda5575bbb7f5c85cd5709e8d2c6c98a24f31caac40d9b5ee6ff2"}
Nov 02 13:09:35.489905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.489891    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:35.494053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.494025    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-27twq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused" start-of-body=
Nov 02 13:09:35.494208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.494075    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podUID="c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": dial tcp 10.134.1.8:15021: connect: connection refused"
Nov 02 13:09:35.536240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.534943    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2" podStartSLOduration=2.534925947 podStartE2EDuration="2.534925947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:35.533856057 +0000 UTC m=+2521.640558909" watchObservedRunningTime="2026-11-02 13:09:35.534925947 +0000 UTC m=+2521.641628803"
Nov 02 13:09:35.617189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.617082    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"]
Nov 02 13:09:35.627048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.624750    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"]
Nov 02 13:09:35.636540 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.631191    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:35.702501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.702471    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:09:35.728093 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.728052    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:35.792754 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.792716    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-routing-overlay-volume\") pod \"f4217d62-9037-41fd-9279-5c59120431a3\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") "
Nov 02 13:09:35.792976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.792858    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-kube-api-access-v2967\" (UniqueName: \"kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-kube-api-access-v2967\") pod \"f4217d62-9037-41fd-9279-5c59120431a3\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") "
Nov 02 13:09:35.792976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.792903    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-provider-credentials\") pod \"f4217d62-9037-41fd-9279-5c59120431a3\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") "
Nov 02 13:09:35.793159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.792980    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-plugins-config-volume\") pod \"f4217d62-9037-41fd-9279-5c59120431a3\" (UID: \"f4217d62-9037-41fd-9279-5c59120431a3\") "
Nov 02 13:09:35.793969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.793631    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-routing-overlay-volume" pod "f4217d62-9037-41fd-9279-5c59120431a3" (UID: "f4217d62-9037-41fd-9279-5c59120431a3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:35.794125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.793967    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:35.794424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.794404    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-plugins-config-volume" pod "f4217d62-9037-41fd-9279-5c59120431a3" (UID: "f4217d62-9037-41fd-9279-5c59120431a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:35.797400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.797369    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-provider-credentials" pod "f4217d62-9037-41fd-9279-5c59120431a3" (UID: "f4217d62-9037-41fd-9279-5c59120431a3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:35.797627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.797605    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-kube-api-access-v2967" pod "f4217d62-9037-41fd-9279-5c59120431a3" (UID: "f4217d62-9037-41fd-9279-5c59120431a3"). InnerVolumeSpecName "kube-api-access-v2967". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:35.866675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.866637    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:35.895872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.895779    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2967\" (UniqueName: \"kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-kube-api-access-v2967\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:35.895872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.895816    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4217d62-9037-41fd-9279-5c59120431a3-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:35.895872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:35.895826    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4217d62-9037-41fd-9279-5c59120431a3-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:36.061018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:36.045546    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:36.509557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:36.509526    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"
Nov 02 13:09:36.509860 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:36.509553    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55" event={"ID":"f4217d62-9037-41fd-9279-5c59120431a3","Type":"ContainerDied","Data":"22db9384543d77defb4783dedcdd36d745ac77ecfd047f77cff71d87e1cef19b"}
Nov 02 13:09:36.509986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:36.509914    5879 scope.go:122] "RemoveContainer" containerID="8d741c4a9e9d9969505687746a87cfc174bfbf5a77077ff52046e544f7dbf9ce"
Nov 02 13:09:36.574034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:36.572774    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"]
Nov 02 13:09:36.588127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:36.588090    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55"]
Nov 02 13:09:36.690480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:36.690369    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6f96cb5db5-7kr55" podUID="f4217d62-9037-41fd-9279-5c59120431a3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.247:50052\" within 1s: context deadline exceeded"
Nov 02 13:09:36.690480 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:36.690469    5879 logging.go:55] [core] [Channel #5320 SubChannel #5321]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.247:50052", ServerName: "10.134.0.247:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.247:50052: operation was canceled"
Nov 02 13:09:37.352566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:37.352528    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:37.447965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:37.447930    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:37.448447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:37.448411    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:37.449273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:37.449257    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq"
Nov 02 13:09:37.487717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:37.487664    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:37.726405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:37.726369    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:37.882167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:37.882132    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:38.005025 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:38.004904    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:38.018368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:38.018335    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4217d62-9037-41fd-9279-5c59120431a3" path="/var/lib/kubelet/pods/f4217d62-9037-41fd-9279-5c59120431a3/volumes"
Nov 02 13:09:38.034623 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:38.034585    5879 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-433df49ba6f6: secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:38.034623 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:38.034620    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t: secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:38.034864 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:38.034681    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials podName:ab1d0069-ae90-4f41-94f2-739840f6a71d nodeName:}" failed. No retries permitted until 2026-11-02 13:09:46.034660364 +0000 UTC m=+2532.141363204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials") pod "payload-processing-external-model-6b4d54cfdc-cph7t" (UID: "ab1d0069-ae90-4f41-94f2-739840f6a71d") : secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:38.132304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:38.132275    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:38.463094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:38.463059    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:38.592048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:38.592012    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:38.762747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:38.762647    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt" podUID="5ea67639-2f9f-41c7-bdc2-ab599a384285" containerName="payload-processing" containerID="cri-o://c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea" gracePeriod=25
Nov 02 13:09:38.927241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:38.927203    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"]
Nov 02 13:09:39.002017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:38.980530    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"]
Nov 02 13:09:39.002017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:38.982206    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f4217d62-9037-41fd-9279-5c59120431a3" containerName="payload-processing"
Nov 02 13:09:39.002017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:38.982247    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4217d62-9037-41fd-9279-5c59120431a3" containerName="payload-processing"
Nov 02 13:09:39.002017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:38.982477    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4217d62-9037-41fd-9279-5c59120431a3" containerName="payload-processing"
Nov 02 13:09:39.002017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:38.989657    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.019108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.015208    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"]
Nov 02 13:09:39.060471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.060333    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-plugins-config-volume\") pod \"payload-processing-external-model-7bb658f5fc-8gkrv\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.060471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.060379    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-routing-overlay-volume\") pod \"payload-processing-external-model-7bb658f5fc-8gkrv\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.060735 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.060548    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdm6m\" (UniqueName: \"kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-kube-api-access-xdm6m\") pod \"payload-processing-external-model-7bb658f5fc-8gkrv\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.061206 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.061019    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-provider-credentials\") pod \"payload-processing-external-model-7bb658f5fc-8gkrv\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.160538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.160506    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:39.165040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.163745    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-plugins-config-volume\") pod \"payload-processing-external-model-7bb658f5fc-8gkrv\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.165040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.163794    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-routing-overlay-volume\") pod \"payload-processing-external-model-7bb658f5fc-8gkrv\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.165040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.163923    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdm6m\" (UniqueName: \"kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-kube-api-access-xdm6m\") pod \"payload-processing-external-model-7bb658f5fc-8gkrv\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.165040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.164141    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-provider-credentials\") pod \"payload-processing-external-model-7bb658f5fc-8gkrv\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.165526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.165479    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-routing-overlay-volume\") pod \"payload-processing-external-model-7bb658f5fc-8gkrv\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.185020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.166274    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-plugins-config-volume\") pod \"payload-processing-external-model-7bb658f5fc-8gkrv\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.185020 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:39.166786    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-90cda9/agc-cred-26e198286e55: secret "agc-cred-26e198286e55" not found
Nov 02 13:09:39.185020 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:39.166806    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-90cda9/agc-cred-8f1d63a606c1: secret "agc-cred-8f1d63a606c1" not found
Nov 02 13:09:39.185020 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:39.166820    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk: [secret "agc-cred-26e198286e55" not found, secret "agc-cred-8f1d63a606c1" not found]
Nov 02 13:09:39.185020 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:39.166889    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials podName:0e8b12d6-622a-49d1-bd56-41a22e7f4415 nodeName:}" failed. No retries permitted until 2026-11-02 13:09:55.166869623 +0000 UTC m=+2541.273572456 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk" (UID: "0e8b12d6-622a-49d1-bd56-41a22e7f4415") : [secret "agc-cred-26e198286e55" not found, secret "agc-cred-8f1d63a606c1" not found]
Nov 02 13:09:39.196019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.190133    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:39.226138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.226026    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-provider-credentials\") pod \"payload-processing-external-model-7bb658f5fc-8gkrv\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.266487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.265047    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-kube-api-access-vvmj7\" (UniqueName: \"kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-kube-api-access-vvmj7\") pod \"5ea67639-2f9f-41c7-bdc2-ab599a384285\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") "
Nov 02 13:09:39.266487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.265135    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-provider-credentials\") pod \"5ea67639-2f9f-41c7-bdc2-ab599a384285\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") "
Nov 02 13:09:39.266487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.265259    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-plugins-config-volume\") pod \"5ea67639-2f9f-41c7-bdc2-ab599a384285\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") "
Nov 02 13:09:39.266487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.265346    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-routing-overlay-volume\") pod \"5ea67639-2f9f-41c7-bdc2-ab599a384285\" (UID: \"5ea67639-2f9f-41c7-bdc2-ab599a384285\") "
Nov 02 13:09:39.266487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.266314    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-plugins-config-volume" pod "5ea67639-2f9f-41c7-bdc2-ab599a384285" (UID: "5ea67639-2f9f-41c7-bdc2-ab599a384285"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:39.266957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.266559    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-routing-overlay-volume" pod "5ea67639-2f9f-41c7-bdc2-ab599a384285" (UID: "5ea67639-2f9f-41c7-bdc2-ab599a384285"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:39.270509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.270429    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-kube-api-access-vvmj7" pod "5ea67639-2f9f-41c7-bdc2-ab599a384285" (UID: "5ea67639-2f9f-41c7-bdc2-ab599a384285"). InnerVolumeSpecName "kube-api-access-vvmj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:39.270509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.270453    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-provider-credentials" pod "5ea67639-2f9f-41c7-bdc2-ab599a384285" (UID: "5ea67639-2f9f-41c7-bdc2-ab599a384285"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:39.275168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.275135    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdm6m\" (UniqueName: \"kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-kube-api-access-xdm6m\") pod \"payload-processing-external-model-7bb658f5fc-8gkrv\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") " pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.322679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.322614    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:39.335464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.335260    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:39.367157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.367091    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvmj7\" (UniqueName: \"kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-kube-api-access-vvmj7\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:39.367157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.367120    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5ea67639-2f9f-41c7-bdc2-ab599a384285-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:39.367157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.367129    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:39.367157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.367138    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5ea67639-2f9f-41c7-bdc2-ab599a384285-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:39.620028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.618814    5879 generic.go:356] "Generic (PLEG): container finished" podID="5ea67639-2f9f-41c7-bdc2-ab599a384285" containerID="c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea" exitCode=0
Nov 02 13:09:39.620028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.619200    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt" event={"ID":"5ea67639-2f9f-41c7-bdc2-ab599a384285","Type":"ContainerDied","Data":"c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea"}
Nov 02 13:09:39.620028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.619261    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt" event={"ID":"5ea67639-2f9f-41c7-bdc2-ab599a384285","Type":"ContainerDied","Data":"0fd482c904b68625a89725d93d26f0f4032f59521a428ced197bbbd60f50bbf6"}
Nov 02 13:09:39.620028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.619293    5879 scope.go:122] "RemoveContainer" containerID="c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea"
Nov 02 13:09:39.620028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.619496    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"
Nov 02 13:09:39.762153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.755250    5879 scope.go:122] "RemoveContainer" containerID="c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea"
Nov 02 13:09:39.762153 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:39.755790    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea\": container with ID starting with c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea not found: ID does not exist" containerID="c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea"
Nov 02 13:09:39.762153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.755823    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea"} err="failed to get container status \"c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea\": rpc error: code = NotFound desc = could not find container \"c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea\": container with ID starting with c2e9369570e2f4b35a5cf6fcc1e293970080360af4adebb953b8496a4c17a6ea not found: ID does not exist"
Nov 02 13:09:39.804018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.803614    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"]
Nov 02 13:09:39.817266 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:39.810374    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc67eaf_0763_4be6_8db7_a52d2ae14a22.slice/crio-e7fffa2ccea7158b6d2997dedcca931096f389bded8cd490baf1086b1d3fb33d WatchSource:0}: Error finding container e7fffa2ccea7158b6d2997dedcca931096f389bded8cd490baf1086b1d3fb33d: Status 404 returned error can't find the container with id e7fffa2ccea7158b6d2997dedcca931096f389bded8cd490baf1086b1d3fb33d
Nov 02 13:09:39.817266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.812072    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-c8nzt"]
Nov 02 13:09:39.818648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:39.818117    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"]
Nov 02 13:09:40.025702 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.025665    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea67639-2f9f-41c7-bdc2-ab599a384285" path="/var/lib/kubelet/pods/5ea67639-2f9f-41c7-bdc2-ab599a384285/volumes"
Nov 02 13:09:40.299537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.299433    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:40.620023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.619177    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk" podUID="0e8b12d6-622a-49d1-bd56-41a22e7f4415" containerName="payload-processing" containerID="cri-o://57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c" gracePeriod=25
Nov 02 13:09:40.630727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.627509    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57" podUID="7b8c6740-cb02-4212-975b-8203ef99d355" containerName="payload-processing" containerID="cri-o://c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4" gracePeriod=25
Nov 02 13:09:40.643395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.643358    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:40.691685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.691640    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv" event={"ID":"6fc67eaf-0763-4be6-8db7-a52d2ae14a22","Type":"ContainerStarted","Data":"dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5"}
Nov 02 13:09:40.698430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.698400    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:40.698666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.698646    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv" event={"ID":"6fc67eaf-0763-4be6-8db7-a52d2ae14a22","Type":"ContainerStarted","Data":"e7fffa2ccea7158b6d2997dedcca931096f389bded8cd490baf1086b1d3fb33d"}
Nov 02 13:09:40.701687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.701656    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"]
Nov 02 13:09:40.703424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.703394    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5ea67639-2f9f-41c7-bdc2-ab599a384285" containerName="payload-processing"
Nov 02 13:09:40.703424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.703425    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea67639-2f9f-41c7-bdc2-ab599a384285" containerName="payload-processing"
Nov 02 13:09:40.703883 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.703862    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea67639-2f9f-41c7-bdc2-ab599a384285" containerName="payload-processing"
Nov 02 13:09:40.708936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.708916    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:40.713031 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.712963    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"]
Nov 02 13:09:40.727359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.727306    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:09:40.727359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.727313    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:09:40.795039 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.790446    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv" podStartSLOduration=2.790423084 podStartE2EDuration="2.790423084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:40.740619931 +0000 UTC m=+2526.847322784" watchObservedRunningTime="2026-11-02 13:09:40.790423084 +0000 UTC m=+2526.897125937"
Nov 02 13:09:40.808262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.808221    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c984db05-067d-4977-be7d-da62869b388f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-513775d4-5j8p6\" (UID: \"c984db05-067d-4977-be7d-da62869b388f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:40.808726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.808482    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c984db05-067d-4977-be7d-da62869b388f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-513775d4-5j8p6\" (UID: \"c984db05-067d-4977-be7d-da62869b388f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:40.808726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.808682    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fq2b\" (UniqueName: \"kubernetes.io/projected/c984db05-067d-4977-be7d-da62869b388f-kube-api-access-8fq2b\") pod \"maas-api-revoke-sub-513775d4-5j8p6\" (UID: \"c984db05-067d-4977-be7d-da62869b388f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:40.910295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.910254    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c984db05-067d-4977-be7d-da62869b388f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-513775d4-5j8p6\" (UID: \"c984db05-067d-4977-be7d-da62869b388f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:40.910459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.910412    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fq2b\" (UniqueName: \"kubernetes.io/projected/c984db05-067d-4977-be7d-da62869b388f-kube-api-access-8fq2b\") pod \"maas-api-revoke-sub-513775d4-5j8p6\" (UID: \"c984db05-067d-4977-be7d-da62869b388f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:40.910528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.910465    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c984db05-067d-4977-be7d-da62869b388f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-513775d4-5j8p6\" (UID: \"c984db05-067d-4977-be7d-da62869b388f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:40.911967 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.911934    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c984db05-067d-4977-be7d-da62869b388f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-513775d4-5j8p6\" (UID: \"c984db05-067d-4977-be7d-da62869b388f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:40.916412 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.916375    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c984db05-067d-4977-be7d-da62869b388f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-513775d4-5j8p6\" (UID: \"c984db05-067d-4977-be7d-da62869b388f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:40.937557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.937525    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fq2b\" (UniqueName: \"kubernetes.io/projected/c984db05-067d-4977-be7d-da62869b388f-kube-api-access-8fq2b\") pod \"maas-api-revoke-sub-513775d4-5j8p6\" (UID: \"c984db05-067d-4977-be7d-da62869b388f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:40.980383 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.980354    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:40.988585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:40.988563    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:41.010026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.009972    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"]
Nov 02 13:09:41.011703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.011676    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-plugins-config-volume\") pod \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") "
Nov 02 13:09:41.011852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.011803    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-plugins-config-volume\") pod \"7b8c6740-cb02-4212-975b-8203ef99d355\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") "
Nov 02 13:09:41.011934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.011850    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials\") pod \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") "
Nov 02 13:09:41.011934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.011915    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-kube-api-access-x42cm\" (UniqueName: \"kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-kube-api-access-x42cm\") pod \"7b8c6740-cb02-4212-975b-8203ef99d355\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") "
Nov 02 13:09:41.012058 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.011980    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-routing-overlay-volume\") pod \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") "
Nov 02 13:09:41.012744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.012190    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-plugins-config-volume" pod "0e8b12d6-622a-49d1-bd56-41a22e7f4415" (UID: "0e8b12d6-622a-49d1-bd56-41a22e7f4415"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:41.012744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.012223    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-kube-api-access-c66t4\" (UniqueName: \"kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-kube-api-access-c66t4\") pod \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\" (UID: \"0e8b12d6-622a-49d1-bd56-41a22e7f4415\") "
Nov 02 13:09:41.012744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.012279    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-provider-credentials\") pod \"7b8c6740-cb02-4212-975b-8203ef99d355\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") "
Nov 02 13:09:41.012744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.012314    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-routing-overlay-volume\") pod \"7b8c6740-cb02-4212-975b-8203ef99d355\" (UID: \"7b8c6740-cb02-4212-975b-8203ef99d355\") "
Nov 02 13:09:41.012744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.012616    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-routing-overlay-volume" pod "0e8b12d6-622a-49d1-bd56-41a22e7f4415" (UID: "0e8b12d6-622a-49d1-bd56-41a22e7f4415"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:41.012744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.012636    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-plugins-config-volume" pod "7b8c6740-cb02-4212-975b-8203ef99d355" (UID: "7b8c6740-cb02-4212-975b-8203ef99d355"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:41.013188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.013162    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:41.013288 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.013192    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:41.013288 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.013207    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e8b12d6-622a-49d1-bd56-41a22e7f4415-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:41.013752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.013711    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-routing-overlay-volume" pod "7b8c6740-cb02-4212-975b-8203ef99d355" (UID: "7b8c6740-cb02-4212-975b-8203ef99d355"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:41.014188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.014168    5879 cpu_manager.go:414] "containerMap: removing container" podUID="0e8b12d6-622a-49d1-bd56-41a22e7f4415" containerName="payload-processing"
Nov 02 13:09:41.014370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.014355    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8b12d6-622a-49d1-bd56-41a22e7f4415" containerName="payload-processing"
Nov 02 13:09:41.014519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.014505    5879 cpu_manager.go:414] "containerMap: removing container" podUID="7b8c6740-cb02-4212-975b-8203ef99d355" containerName="payload-processing"
Nov 02 13:09:41.015140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.015121    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8c6740-cb02-4212-975b-8203ef99d355" containerName="payload-processing"
Nov 02 13:09:41.016577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.016560    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8b12d6-622a-49d1-bd56-41a22e7f4415" containerName="payload-processing"
Nov 02 13:09:41.016707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.016695    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8c6740-cb02-4212-975b-8203ef99d355" containerName="payload-processing"
Nov 02 13:09:41.019568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.018203    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-kube-api-access-x42cm" pod "7b8c6740-cb02-4212-975b-8203ef99d355" (UID: "7b8c6740-cb02-4212-975b-8203ef99d355"). InnerVolumeSpecName "kube-api-access-x42cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:41.019568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.018352    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-kube-api-access-c66t4" pod "0e8b12d6-622a-49d1-bd56-41a22e7f4415" (UID: "0e8b12d6-622a-49d1-bd56-41a22e7f4415"). InnerVolumeSpecName "kube-api-access-c66t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:41.027160 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.025407    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials" pod "0e8b12d6-622a-49d1-bd56-41a22e7f4415" (UID: "0e8b12d6-622a-49d1-bd56-41a22e7f4415"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:41.027730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.027712    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.046101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.045588    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"kube-root-ca.crt\""
Nov 02 13:09:41.046101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.045765    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"payload-processing-external-model-e2e--9bdd4eed-dockercfg-vvx5k\""
Nov 02 13:09:41.046101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.045844    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"agc-cred-ae68aaa00e1f\""
Nov 02 13:09:41.046101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.045922    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"payload-processing-external-model-plugins-e2e-extmod-a-035dac\""
Nov 02 13:09:41.046101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.046055    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"openshift-service-ca.crt\""
Nov 02 13:09:41.046485 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.046171    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"routing-overlay\""
Nov 02 13:09:41.104155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.104098    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-provider-credentials" pod "7b8c6740-cb02-4212-975b-8203ef99d355" (UID: "7b8c6740-cb02-4212-975b-8203ef99d355"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:41.105423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.105333    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:41.107231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.107211    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"]
Nov 02 13:09:41.114274 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.114226    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxlb8\" (UniqueName: \"kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-kube-api-access-jxlb8\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.114489 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.114464    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.114955 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.114932    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.115242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.115222    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.115828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.115810    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c66t4\" (UniqueName: \"kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-kube-api-access-c66t4\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:41.116129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.116114    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:41.116363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.116346    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b8c6740-cb02-4212-975b-8203ef99d355-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:41.116478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.116466    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e8b12d6-622a-49d1-bd56-41a22e7f4415-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:41.116559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.116549    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x42cm\" (UniqueName: \"kubernetes.io/projected/7b8c6740-cb02-4212-975b-8203ef99d355-kube-api-access-x42cm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:41.221043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.218247    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.221043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.218394    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.221043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.218463    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlb8\" (UniqueName: \"kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-kube-api-access-jxlb8\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.221043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.218546    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.221043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.220023    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.221043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.220650    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.224028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.223972    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.245696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.245666    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlb8\" (UniqueName: \"kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-kube-api-access-jxlb8\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.359606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.359564    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:41.364268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.364171    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:41.433884 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.433852    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"]
Nov 02 13:09:41.434404 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:41.434374    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc984db05_067d_4977_be7d_da62869b388f.slice/crio-6dba5ace31b8a07a05576365ea75f721a1a8139e39542f75b48863b845a114f7 WatchSource:0}: Error finding container 6dba5ace31b8a07a05576365ea75f721a1a8139e39542f75b48863b845a114f7: Status 404 returned error can't find the container with id 6dba5ace31b8a07a05576365ea75f721a1a8139e39542f75b48863b845a114f7
Nov 02 13:09:41.505837 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.505802    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:41.663614 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.663567    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:41.679975 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.679209    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"]
Nov 02 13:09:41.683297 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:41.683261    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6522c4_39f2_4057_9e66_594e92c504a3.slice/crio-465172e8ca361ce24c590ae726920b329e7ce0fcaf05d1f8538b9090a6ba40c1 WatchSource:0}: Error finding container 465172e8ca361ce24c590ae726920b329e7ce0fcaf05d1f8538b9090a6ba40c1: Status 404 returned error can't find the container with id 465172e8ca361ce24c590ae726920b329e7ce0fcaf05d1f8538b9090a6ba40c1
Nov 02 13:09:41.703065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.702634    5879 generic.go:356] "Generic (PLEG): container finished" podID="0e8b12d6-622a-49d1-bd56-41a22e7f4415" containerID="57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c" exitCode=0
Nov 02 13:09:41.703065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.702686    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"
Nov 02 13:09:41.703065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.702809    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk" event={"ID":"0e8b12d6-622a-49d1-bd56-41a22e7f4415","Type":"ContainerDied","Data":"57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c"}
Nov 02 13:09:41.703065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.702865    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk" event={"ID":"0e8b12d6-622a-49d1-bd56-41a22e7f4415","Type":"ContainerDied","Data":"186412b5d533f40569626a82395ecd966481619551cb8378630a2eb29ab94731"}
Nov 02 13:09:41.703065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.702896    5879 scope.go:122] "RemoveContainer" containerID="57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c"
Nov 02 13:09:41.704724 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.704701    5879 generic.go:356] "Generic (PLEG): container finished" podID="7b8c6740-cb02-4212-975b-8203ef99d355" containerID="c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4" exitCode=0
Nov 02 13:09:41.704858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.704844    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"
Nov 02 13:09:41.705111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.705089    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57" event={"ID":"7b8c6740-cb02-4212-975b-8203ef99d355","Type":"ContainerDied","Data":"c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4"}
Nov 02 13:09:41.705183 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.705148    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57" event={"ID":"7b8c6740-cb02-4212-975b-8203ef99d355","Type":"ContainerDied","Data":"9867048fa524038050e979f3430d89d81bf9fdf574295189f91eceade9a50672"}
Nov 02 13:09:41.723202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.723132    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6" event={"ID":"c984db05-067d-4977-be7d-da62869b388f","Type":"ContainerStarted","Data":"6dba5ace31b8a07a05576365ea75f721a1a8139e39542f75b48863b845a114f7"}
Nov 02 13:09:41.728267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.728244    5879 scope.go:122] "RemoveContainer" containerID="57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c"
Nov 02 13:09:41.728552 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.728526    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf" event={"ID":"4c6522c4-39f2-4057-9e66-594e92c504a3","Type":"ContainerStarted","Data":"465172e8ca361ce24c590ae726920b329e7ce0fcaf05d1f8538b9090a6ba40c1"}
Nov 02 13:09:41.729266 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:41.729243    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c\": container with ID starting with 57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c not found: ID does not exist" containerID="57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c"
Nov 02 13:09:41.729377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.729275    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c"} err="failed to get container status \"57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c\": rpc error: code = NotFound desc = could not find container \"57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c\": container with ID starting with 57516a580d7a8c9a2637a0c2efcb6e71484e258ee2152052f655f3fd353f3f2c not found: ID does not exist"
Nov 02 13:09:41.729377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.729297    5879 scope.go:122] "RemoveContainer" containerID="c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4"
Nov 02 13:09:41.748111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.748070    5879 scope.go:122] "RemoveContainer" containerID="c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4"
Nov 02 13:09:41.748516 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:41.748493    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4\": container with ID starting with c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4 not found: ID does not exist" containerID="c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4"
Nov 02 13:09:41.748629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.748530    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4"} err="failed to get container status \"c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4\": rpc error: code = NotFound desc = could not find container \"c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4\": container with ID starting with c2c856b434daf91484fc45af0411e45e3a2fa6430588be2e1ec1f011edaae5a4 not found: ID does not exist"
Nov 02 13:09:41.785059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.784984    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"]
Nov 02 13:09:41.789078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.789047    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-8fdfmm5pk"]
Nov 02 13:09:41.825309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.825277    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"]
Nov 02 13:09:41.829462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.829433    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-578fglj57"]
Nov 02 13:09:41.846371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.846326    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:41.985080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:41.981915    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:42.027389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.027352    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8b12d6-622a-49d1-bd56-41a22e7f4415" path="/var/lib/kubelet/pods/0e8b12d6-622a-49d1-bd56-41a22e7f4415/volumes"
Nov 02 13:09:42.027871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.027855    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8c6740-cb02-4212-975b-8203ef99d355" path="/var/lib/kubelet/pods/7b8c6740-cb02-4212-975b-8203ef99d355/volumes"
Nov 02 13:09:42.086065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.086023    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"]
Nov 02 13:09:42.092155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.092125    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.108560 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.108529    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"openshift-service-ca.crt\""
Nov 02 13:09:42.108720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.108631    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"routing-overlay\""
Nov 02 13:09:42.108720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.108685    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"kube-root-ca.crt\""
Nov 02 13:09:42.108720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.108711    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"agc-cred-2419caaf5f1f\""
Nov 02 13:09:42.108904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.108734    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"payload-processing-external-model-plugins-e2e-extmod-b-00d004\""
Nov 02 13:09:42.108904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.108870    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"payload-processing-external-model-e2e--8054d3fe-dockercfg-c4r78\""
Nov 02 13:09:42.112878 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.112853    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"]
Nov 02 13:09:42.132642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.131953    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.132642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.132112    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.132642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.132257    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58hn6\" (UniqueName: \"kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-kube-api-access-58hn6\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.132642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.132463    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.225946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.225913    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"]
Nov 02 13:09:42.230769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.230743    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.233933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.233903    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58hn6\" (UniqueName: \"kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-kube-api-access-58hn6\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.234113 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.234065    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.234218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.234175    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.234218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.234201    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.235113 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.235054    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.235226 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.235150    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.236903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.236883    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.279362 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.279327    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58hn6\" (UniqueName: \"kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-kube-api-access-58hn6\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.282275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.282133    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"]
Nov 02 13:09:42.344827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.344720    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.345170 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.345148    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.345356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.345331    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.345536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.345517    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.345692 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.345675    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e95cd7-e5a6-446f-90c5-db47c0efe379-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.345874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.345855    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.348508 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.348473    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.348747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.348727    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.348939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.348921    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv68r\" (UniqueName: \"kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-kube-api-access-sv68r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.408333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.408291    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:42.408977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.408955    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:42.452900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.451566    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.452900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.451631    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.452900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.451678    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.452900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.451714    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.452900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.451749    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e95cd7-e5a6-446f-90c5-db47c0efe379-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.452900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.451781    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.452900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.451875    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.452900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.451907    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.452900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.451939    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv68r\" (UniqueName: \"kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-kube-api-access-sv68r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.452900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.452437    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.452900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.452550    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.453940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.452985    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.453940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.453155    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e95cd7-e5a6-446f-90c5-db47c0efe379-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.453940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.453394    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.455835 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.455809    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.457716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.457689    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.477948 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.477913    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.478359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.478333    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv68r\" (UniqueName: \"kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-kube-api-access-sv68r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-sg8t7\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.620297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.620255    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:42.749126 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.749035    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"]
Nov 02 13:09:42.759256 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:42.759231    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc6bf3f_d673_4095_bf6c_8a81b66fe351.slice/crio-643f64f30ca6cfd19c91ea49de439e839ec523c524b20de9d5cb359e101615ab WatchSource:0}: Error finding container 643f64f30ca6cfd19c91ea49de439e839ec523c524b20de9d5cb359e101615ab: Status 404 returned error can't find the container with id 643f64f30ca6cfd19c91ea49de439e839ec523c524b20de9d5cb359e101615ab
Nov 02 13:09:42.759779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.759747    5879 generic.go:356] "Generic (PLEG): container finished" podID="c984db05-067d-4977-be7d-da62869b388f" containerID="1607a7376542c7fa1199e5dcfda89f3de6c05b1f61543565747b6b9c33f9fb1f" exitCode=0
Nov 02 13:09:42.760031 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.759949    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6" event={"ID":"c984db05-067d-4977-be7d-da62869b388f","Type":"ContainerDied","Data":"1607a7376542c7fa1199e5dcfda89f3de6c05b1f61543565747b6b9c33f9fb1f"}
Nov 02 13:09:42.764143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.763984    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf" event={"ID":"4c6522c4-39f2-4057-9e66-594e92c504a3","Type":"ContainerStarted","Data":"b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163"}
Nov 02 13:09:42.764143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.764081    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:42.903238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.903175    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf" podStartSLOduration=2.903159053 podStartE2EDuration="2.903159053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:42.897846972 +0000 UTC m=+2529.004549834" watchObservedRunningTime="2026-11-02 13:09:42.903159053 +0000 UTC m=+2529.009861904"
Nov 02 13:09:42.957810 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:42.957769    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e95cd7_e5a6_446f_90c5_db47c0efe379.slice/crio-b955bdbc30598258195b4809c3d36326d2c2d46c43cdff16036c87f37bb337b2 WatchSource:0}: Error finding container b955bdbc30598258195b4809c3d36326d2c2d46c43cdff16036c87f37bb337b2: Status 404 returned error can't find the container with id b955bdbc30598258195b4809c3d36326d2c2d46c43cdff16036c87f37bb337b2
Nov 02 13:09:42.960664 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.960636    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"]
Nov 02 13:09:42.961063 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.961025    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:09:42.961130 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.961113    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:09:42.961184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:42.961156    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:09:43.277071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:43.275591    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:43.405164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:43.405129    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:43.775940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:43.775905    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr" event={"ID":"dcc6bf3f-d673-4095-bf6c-8a81b66fe351","Type":"ContainerStarted","Data":"d9c6509e1b5d56e145628f8cd37641a6f865e8e5d111fe871bc7605e2c5e0869"}
Nov 02 13:09:43.776271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:43.776239    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr" event={"ID":"dcc6bf3f-d673-4095-bf6c-8a81b66fe351","Type":"ContainerStarted","Data":"643f64f30ca6cfd19c91ea49de439e839ec523c524b20de9d5cb359e101615ab"}
Nov 02 13:09:43.776401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:43.776281    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:43.778221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:43.778188    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" event={"ID":"31e95cd7-e5a6-446f-90c5-db47c0efe379","Type":"ContainerStarted","Data":"e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7"}
Nov 02 13:09:43.778348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:43.778240    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" event={"ID":"31e95cd7-e5a6-446f-90c5-db47c0efe379","Type":"ContainerStarted","Data":"b955bdbc30598258195b4809c3d36326d2c2d46c43cdff16036c87f37bb337b2"}
Nov 02 13:09:43.834565 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:43.834495    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr" podStartSLOduration=1.834476555 podStartE2EDuration="1.834476555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:43.830261397 +0000 UTC m=+2529.936964241" watchObservedRunningTime="2026-11-02 13:09:43.834476555 +0000 UTC m=+2529.941179408"
Nov 02 13:09:43.914438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:43.914384    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" podStartSLOduration=1.914370927 podStartE2EDuration="1.914370927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:43.913620362 +0000 UTC m=+2530.020323215" watchObservedRunningTime="2026-11-02 13:09:43.914370927 +0000 UTC m=+2530.021073777"
Nov 02 13:09:43.931717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:43.931112    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2" podUID="ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d" containerName="payload-processing" containerID="cri-o://01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691" gracePeriod=25
Nov 02 13:09:43.952458 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:43.952418    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:44.041106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.038925    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:44.104018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.081405    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c984db05-067d-4977-be7d-da62869b388f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c984db05-067d-4977-be7d-da62869b388f-maas-api-xks-ca\") pod \"c984db05-067d-4977-be7d-da62869b388f\" (UID: \"c984db05-067d-4977-be7d-da62869b388f\") "
Nov 02 13:09:44.104018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.081485    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c984db05-067d-4977-be7d-da62869b388f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c984db05-067d-4977-be7d-da62869b388f-maas-api-openshift-ca\") pod \"c984db05-067d-4977-be7d-da62869b388f\" (UID: \"c984db05-067d-4977-be7d-da62869b388f\") "
Nov 02 13:09:44.104018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.081665    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c984db05-067d-4977-be7d-da62869b388f-kube-api-access-8fq2b\" (UniqueName: \"kubernetes.io/projected/c984db05-067d-4977-be7d-da62869b388f-kube-api-access-8fq2b\") pod \"c984db05-067d-4977-be7d-da62869b388f\" (UID: \"c984db05-067d-4977-be7d-da62869b388f\") "
Nov 02 13:09:44.113623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.113530    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:44.116268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.116231    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c984db05-067d-4977-be7d-da62869b388f-maas-api-openshift-ca" pod "c984db05-067d-4977-be7d-da62869b388f" (UID: "c984db05-067d-4977-be7d-da62869b388f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:44.120973 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.120932    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c984db05-067d-4977-be7d-da62869b388f-kube-api-access-8fq2b" pod "c984db05-067d-4977-be7d-da62869b388f" (UID: "c984db05-067d-4977-be7d-da62869b388f"). InnerVolumeSpecName "kube-api-access-8fq2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:44.122935 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.122906    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c984db05-067d-4977-be7d-da62869b388f-maas-api-xks-ca" pod "c984db05-067d-4977-be7d-da62869b388f" (UID: "c984db05-067d-4977-be7d-da62869b388f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:09:44.194095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.189729    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fq2b\" (UniqueName: \"kubernetes.io/projected/c984db05-067d-4977-be7d-da62869b388f-kube-api-access-8fq2b\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:44.194095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.189774    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c984db05-067d-4977-be7d-da62869b388f-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:44.194095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.189791    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c984db05-067d-4977-be7d-da62869b388f-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:44.317718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.317634    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:44.419115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.419078    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-routing-overlay-volume\") pod \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") "
Nov 02 13:09:44.419349 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.419152    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-plugins-config-volume\") pod \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") "
Nov 02 13:09:44.419349 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.419296    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-provider-credentials\") pod \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") "
Nov 02 13:09:44.419349 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.419341    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-kube-api-access-kwmd7\" (UniqueName: \"kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-kube-api-access-kwmd7\") pod \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\" (UID: \"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d\") "
Nov 02 13:09:44.420048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.419981    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-plugins-config-volume" pod "ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d" (UID: "ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:44.420342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.420322    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-routing-overlay-volume" pod "ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d" (UID: "ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:44.427217 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.427112    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-kube-api-access-kwmd7" pod "ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d" (UID: "ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d"). InnerVolumeSpecName "kube-api-access-kwmd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:44.428206 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.428182    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-provider-credentials" pod "ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d" (UID: "ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:44.521403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.521372    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:44.521403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.521398    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:44.521403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.521409    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:44.521650 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.521420    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwmd7\" (UniqueName: \"kubernetes.io/projected/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d-kube-api-access-kwmd7\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:44.621342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.621299    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:44.623193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.623167    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sg8t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:09:44.623348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.623215    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 13:09:44.708369 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.708325    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:44.810125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.809976    5879 generic.go:356] "Generic (PLEG): container finished" podID="ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d" containerID="01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691" exitCode=0
Nov 02 13:09:44.810546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.810414    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2" event={"ID":"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d","Type":"ContainerDied","Data":"01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691"}
Nov 02 13:09:44.810546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.810424    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"
Nov 02 13:09:44.810546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.810489    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2" event={"ID":"ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d","Type":"ContainerDied","Data":"5766a96011aeda5575bbb7f5c85cd5709e8d2c6c98a24f31caac40d9b5ee6ff2"}
Nov 02 13:09:44.810546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.810523    5879 scope.go:122] "RemoveContainer" containerID="01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691"
Nov 02 13:09:44.812742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.812716    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"
Nov 02 13:09:44.812853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.812731    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6" event={"ID":"c984db05-067d-4977-be7d-da62869b388f","Type":"ContainerDied","Data":"6dba5ace31b8a07a05576365ea75f721a1a8139e39542f75b48863b845a114f7"}
Nov 02 13:09:44.812853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.812788    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dba5ace31b8a07a05576365ea75f721a1a8139e39542f75b48863b845a114f7"
Nov 02 13:09:44.822775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.822750    5879 scope.go:122] "RemoveContainer" containerID="01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691"
Nov 02 13:09:44.823214 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:44.823184    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691\": container with ID starting with 01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691 not found: ID does not exist" containerID="01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691"
Nov 02 13:09:44.823293 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.823226    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691"} err="failed to get container status \"01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691\": rpc error: code = NotFound desc = could not find container \"01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691\": container with ID starting with 01f302a101ba896493337cfd2671e73323583b3b641ea5fbea0357423c515691 not found: ID does not exist"
Nov 02 13:09:44.844444 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:44.844412    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:09:44.912703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.912608    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"]
Nov 02 13:09:44.919080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:44.919049    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-666978d5bb-cjbj2"]
Nov 02 13:09:45.621464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:45.621427    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sg8t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:09:45.621672 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:45.621481    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 13:09:46.020343 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:46.020246    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d" path="/var/lib/kubelet/pods/ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d/volumes"
Nov 02 13:09:46.042568 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:46.042530    5879 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-433df49ba6f6: secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:46.042568 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:46.042569    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t: secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:46.042805 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:09:46.042636    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials podName:ab1d0069-ae90-4f41-94f2-739840f6a71d nodeName:}" failed. No retries permitted until 2026-11-02 13:10:02.042615527 +0000 UTC m=+2548.149318355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials") pod "payload-processing-external-model-6b4d54cfdc-cph7t" (UID: "ab1d0069-ae90-4f41-94f2-739840f6a71d") : secret "agc-cred-433df49ba6f6" not found
Nov 02 13:09:46.130461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:46.130406    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:46.295074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:46.294517    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:46.630679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:46.630623    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sg8t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:09:46.631053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:46.631021    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 13:09:46.763137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:46.763093    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:47.204017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.203703    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:47.430888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.430846    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"]
Nov 02 13:09:47.431855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.431829    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d" containerName="payload-processing"
Nov 02 13:09:47.431855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.431853    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d" containerName="payload-processing"
Nov 02 13:09:47.432061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.431897    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c984db05-067d-4977-be7d-da62869b388f" containerName="revoke-keys"
Nov 02 13:09:47.432061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.431906    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c984db05-067d-4977-be7d-da62869b388f" containerName="revoke-keys"
Nov 02 13:09:47.432153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.432119    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c984db05-067d-4977-be7d-da62869b388f" containerName="revoke-keys"
Nov 02 13:09:47.432153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.432145    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab53418b-58ca-4ac7-bfc9-f02cf9ebcb3d" containerName="payload-processing"
Nov 02 13:09:47.438390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.438370    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.456298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.456232    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"]
Nov 02 13:09:47.464303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.464141    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"openshift-service-ca.crt\""
Nov 02 13:09:47.464303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.464142    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"agc-cred-d4cab92a7691\""
Nov 02 13:09:47.464303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.464193    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"kube-root-ca.crt\""
Nov 02 13:09:47.464303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.464257    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"routing-overlay\""
Nov 02 13:09:47.464303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.464271    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-14fcb6\""
Nov 02 13:09:47.568722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.568686    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.568931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.568900    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.569035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.569019    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.569220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.569206    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfdbc\" (UniqueName: \"kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-kube-api-access-kfdbc\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.621697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.621655    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sg8t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:09:47.621862 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.621731    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 13:09:47.670618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.670571    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.670790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.670645    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.670883 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.670857    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdbc\" (UniqueName: \"kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-kube-api-access-kfdbc\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.671269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.671250    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.671639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.671612    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.671877 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.671857    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.674119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.674100    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.714071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.713157    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdbc\" (UniqueName: \"kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-kube-api-access-kfdbc\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:47.763666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:47.763370    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:48.149781 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:48.149749    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a99f05_455d_42be_b61b_f5d7766c79f9.slice/crio-4cbd164733c9a92c7fc8520f7bc8ace78b4cb80935fa39c16f13bdbb046e9f4b WatchSource:0}: Error finding container 4cbd164733c9a92c7fc8520f7bc8ace78b4cb80935fa39c16f13bdbb046e9f4b: Status 404 returned error can't find the container with id 4cbd164733c9a92c7fc8520f7bc8ace78b4cb80935fa39c16f13bdbb046e9f4b
Nov 02 13:09:48.151670 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.151637    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"]
Nov 02 13:09:48.417088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.416400    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"]
Nov 02 13:09:48.422385 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.422356    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.437046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.432219    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"]
Nov 02 13:09:48.438762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.438741    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-035dac\"/\"agc-cred-09724141ab04\""
Nov 02 13:09:48.621502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.621457    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-7577497xp\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.621502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.621467    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sg8t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:09:48.621699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.621526    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 13:09:48.621699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.621649    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-7577497xp\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.621787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.621726    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-7577497xp\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.621787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.621777    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbfm\" (UniqueName: \"kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-kube-api-access-2cbfm\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-7577497xp\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.722890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.722802    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-7577497xp\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.722890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.722858    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbfm\" (UniqueName: \"kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-kube-api-access-2cbfm\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-7577497xp\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.723095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.723026    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-7577497xp\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.723147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.723109    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-7577497xp\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.723516 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.723498    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-7577497xp\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.723745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.723724    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-7577497xp\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.728077 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.726646    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-7577497xp\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.750103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.750073    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbfm\" (UniqueName: \"kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-kube-api-access-2cbfm\") pod \"payload-processing-external-model-e2e-extmod-a-035dac-7577497xp\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") " pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:48.810214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.810175    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:48.867778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.867657    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5" event={"ID":"58a99f05-455d-42be-b61b-f5d7766c79f9","Type":"ContainerStarted","Data":"1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d"}
Nov 02 13:09:48.867778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.867723    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5" event={"ID":"58a99f05-455d-42be-b61b-f5d7766c79f9","Type":"ContainerStarted","Data":"4cbd164733c9a92c7fc8520f7bc8ace78b4cb80935fa39c16f13bdbb046e9f4b"}
Nov 02 13:09:48.867778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.867767    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:09:48.918970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:48.918906    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5" podStartSLOduration=1.918888651 podStartE2EDuration="1.918888651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:48.918275206 +0000 UTC m=+2535.024978057" watchObservedRunningTime="2026-11-02 13:09:48.918888651 +0000 UTC m=+2535.025591502"
Nov 02 13:09:49.042957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.041853    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:49.372955 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.372923    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"]
Nov 02 13:09:49.379394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.379371    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.422040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.415956    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"]
Nov 02 13:09:49.427755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.427603    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-00d004\"/\"agc-cred-9556c2f4a71c\""
Nov 02 13:09:49.433017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.432951    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.433771 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.433261    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.433771 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.433473    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nchqt\" (UniqueName: \"kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-kube-api-access-nchqt\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.433771 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.433502    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.435204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.435137    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"]
Nov 02 13:09:49.535011 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.534947    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.535239 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.535214    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nchqt\" (UniqueName: \"kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-kube-api-access-nchqt\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.535323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.535257    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.535384 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.535350    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.535907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.535881    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.536299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.536273    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.538342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.538311    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.561143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.561106    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nchqt\" (UniqueName: \"kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-kube-api-access-nchqt\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.620763 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.620722    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sg8t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:09:49.620939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.620789    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 13:09:49.752355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.752247    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:49.884430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.883197    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp" event={"ID":"882fb9d3-1c94-4c23-a494-c9b4402d66ac","Type":"ContainerStarted","Data":"9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b"}
Nov 02 13:09:49.884430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.883309    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp" event={"ID":"882fb9d3-1c94-4c23-a494-c9b4402d66ac","Type":"ContainerStarted","Data":"0043b4436ba4c7fc00345c1df4f074d823135d39ff50677702e07dc23f93fb70"}
Nov 02 13:09:49.884430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.883970    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:09:49.929320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:49.929264    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp" podStartSLOduration=1.929249348 podStartE2EDuration="1.929249348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:49.927221966 +0000 UTC m=+2536.033924829" watchObservedRunningTime="2026-11-02 13:09:49.929249348 +0000 UTC m=+2536.035952198"
Nov 02 13:09:50.072606 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:50.072569    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b92183_a743_4f2a_be41_4faa2fc5cbe4.slice/crio-5464359a071d6ee7b11a7085f7187640bdc37975e1ef4f3f7c4ec731128e20db WatchSource:0}: Error finding container 5464359a071d6ee7b11a7085f7187640bdc37975e1ef4f3f7c4ec731128e20db: Status 404 returned error can't find the container with id 5464359a071d6ee7b11a7085f7187640bdc37975e1ef4f3f7c4ec731128e20db
Nov 02 13:09:50.074147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:50.074120    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"]
Nov 02 13:09:50.540478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:50.540418    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:50.621394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:50.621357    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sg8t7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 02 13:09:50.621628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:50.621443    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 02 13:09:50.667338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:50.667302    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:50.834867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:50.834767    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:50.894067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:50.894029    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7" event={"ID":"97b92183-a743-4f2a-be41-4faa2fc5cbe4","Type":"ContainerStarted","Data":"5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03"}
Nov 02 13:09:50.894256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:50.894097    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7" event={"ID":"97b92183-a743-4f2a-be41-4faa2fc5cbe4","Type":"ContainerStarted","Data":"5464359a071d6ee7b11a7085f7187640bdc37975e1ef4f3f7c4ec731128e20db"}
Nov 02 13:09:50.894256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:50.894125    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:09:50.949203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:50.949138    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7" podStartSLOduration=1.949116476 podStartE2EDuration="1.949116476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:50.947443525 +0000 UTC m=+2537.054146399" watchObservedRunningTime="2026-11-02 13:09:50.949116476 +0000 UTC m=+2537.055819330"
Nov 02 13:09:51.002215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:51.002181    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:51.293146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:51.293113    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:51.732229 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:51.732199    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:09:51.826630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:51.826591    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"]
Nov 02 13:09:51.923982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:51.923908    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:52.044770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:52.044640    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:52.455503 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:52.455473    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:52.456572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:52.456538    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:52.463400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:52.463371    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:09:52.809865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:52.809762    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:53.232615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.232579    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:53.479016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.476431    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"]
Nov 02 13:09:53.484362 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.484285    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.502033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.501983    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"agc-cred-0c5fa1a8b1b9\""
Nov 02 13:09:53.502219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.502061    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"]
Nov 02 13:09:53.502808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.502521    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"payload-processing-external-model-e2e--49869bd7-dockercfg-j49vg\""
Nov 02 13:09:53.505636 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.505597    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:53.508277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.508253    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"routing-overlay\""
Nov 02 13:09:53.508408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.508383    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"kube-root-ca.crt\""
Nov 02 13:09:53.508483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.508449    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"openshift-service-ca.crt\""
Nov 02 13:09:53.508545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.508534    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"payload-processing-external-model-plugins-e2e-extmod-a-740f09\""
Nov 02 13:09:53.643971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.643927    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.644233 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.644117    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.644368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.644237    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.644368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.644333    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26x54\" (UniqueName: \"kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-kube-api-access-26x54\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.669591 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.669555    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:53.746302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.746212    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.746478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.746308    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26x54\" (UniqueName: \"kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-kube-api-access-26x54\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.746478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.746426    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.746616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.746585    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.747209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.747176    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.747329 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.747208    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.749269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.749244    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.772442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.772407    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26x54\" (UniqueName: \"kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-kube-api-access-26x54\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:53.782737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.782707    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:09:53.834952 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:53.834905    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:54.069874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.069783    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"]
Nov 02 13:09:54.076254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.076233    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:54.094492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.094466    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:09:54.094630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.094565    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:09:54.094964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.094940    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"]
Nov 02 13:09:54.155969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.155928    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31051485-wrbbl\" (UID: \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:54.156320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.156287    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bv9z\" (UniqueName: \"kubernetes.io/projected/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-kube-api-access-9bv9z\") pod \"maas-api-revoke-sub-31051485-wrbbl\" (UID: \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:54.156470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.156378    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31051485-wrbbl\" (UID: \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:54.174191 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:54.174145    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c066765_c46e_42e1_8d73_75bddd1dce76.slice/crio-2c934749c8aee50c90441bed777f2af36ec07880a4ef85535ce8a04edcc3ab15 WatchSource:0}: Error finding container 2c934749c8aee50c90441bed777f2af36ec07880a4ef85535ce8a04edcc3ab15: Status 404 returned error can't find the container with id 2c934749c8aee50c90441bed777f2af36ec07880a4ef85535ce8a04edcc3ab15
Nov 02 13:09:54.174731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.174702    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"]
Nov 02 13:09:54.258643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.258608    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bv9z\" (UniqueName: \"kubernetes.io/projected/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-kube-api-access-9bv9z\") pod \"maas-api-revoke-sub-31051485-wrbbl\" (UID: \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:54.259705 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.258696    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31051485-wrbbl\" (UID: \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:54.259705 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.258917    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31051485-wrbbl\" (UID: \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:54.260124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.260070    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31051485-wrbbl\" (UID: \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:54.263329 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.263302    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31051485-wrbbl\" (UID: \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:54.284557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.284522    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bv9z\" (UniqueName: \"kubernetes.io/projected/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-kube-api-access-9bv9z\") pod \"maas-api-revoke-sub-31051485-wrbbl\" (UID: \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:54.295241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.295207    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"]
Nov 02 13:09:54.300427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.300402    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.324741 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.324641    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"]
Nov 02 13:09:54.392216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.392178    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:54.463431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.463397    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.463598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.463546    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82vpc\" (UniqueName: \"kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-kube-api-access-82vpc\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.463672 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.463639    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.464094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.464007    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.566082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.566035    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.566368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.566136    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.566368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.566195    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82vpc\" (UniqueName: \"kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-kube-api-access-82vpc\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.566368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.566272    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.567018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.566971    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.567265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.567242    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.569116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.569096    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.591131 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.591060    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82vpc\" (UniqueName: \"kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-kube-api-access-82vpc\") pod \"payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") " pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.615140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.615071    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:54.701147 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:54.701107    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4704a4c4_6a62_4a71_90f2_cba9ede3dd5a.slice/crio-1352d5d8ac8cbb9ebcfd199515bc1b63306f091fabdfcb6c52177b133093f60c WatchSource:0}: Error finding container 1352d5d8ac8cbb9ebcfd199515bc1b63306f091fabdfcb6c52177b133093f60c: Status 404 returned error can't find the container with id 1352d5d8ac8cbb9ebcfd199515bc1b63306f091fabdfcb6c52177b133093f60c
Nov 02 13:09:54.704622 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.704596    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"]
Nov 02 13:09:54.801452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.801381    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:54.817578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.817553    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:09:54.928976 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:54.928933    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77c143f_8739_48c6_b37b_d6226bf36d0f.slice/crio-3a84c4a9078743c8f93eb906d12f35da83a9e90f28e41ca7e796febf23011a1d WatchSource:0}: Error finding container 3a84c4a9078743c8f93eb906d12f35da83a9e90f28e41ca7e796febf23011a1d: Status 404 returned error can't find the container with id 3a84c4a9078743c8f93eb906d12f35da83a9e90f28e41ca7e796febf23011a1d
Nov 02 13:09:54.932038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.931910    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"]
Nov 02 13:09:54.954285 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.954248    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp" event={"ID":"b77c143f-8739-48c6-b37b-d6226bf36d0f","Type":"ContainerStarted","Data":"3a84c4a9078743c8f93eb906d12f35da83a9e90f28e41ca7e796febf23011a1d"}
Nov 02 13:09:54.960579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.960538    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj" event={"ID":"7c066765-c46e-42e1-8d73-75bddd1dce76","Type":"ContainerStarted","Data":"69e4ee0c3c44fb26bb164956b18a208280d4ee1ebbc768b5e7034667e3e21af3"}
Nov 02 13:09:54.960718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.960609    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:09:54.960718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.960627    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj" event={"ID":"7c066765-c46e-42e1-8d73-75bddd1dce76","Type":"ContainerStarted","Data":"2c934749c8aee50c90441bed777f2af36ec07880a4ef85535ce8a04edcc3ab15"}
Nov 02 13:09:54.962339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:54.962308    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl" event={"ID":"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a","Type":"ContainerStarted","Data":"1352d5d8ac8cbb9ebcfd199515bc1b63306f091fabdfcb6c52177b133093f60c"}
Nov 02 13:09:55.008542 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:55.008492    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj" podStartSLOduration=2.008478162 podStartE2EDuration="2.008478162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:55.00469022 +0000 UTC m=+2541.111393072" watchObservedRunningTime="2026-11-02 13:09:55.008478162 +0000 UTC m=+2541.115181012"
Nov 02 13:09:55.389107 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:55.389076    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:55.973146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:55.973104    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl" event={"ID":"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a","Type":"ContainerStarted","Data":"5cbe265c473543de5843a024cc0d6ffae7c01172204afdd2f39bea8a124b39f8"}
Nov 02 13:09:55.975054 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:55.975021    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp" event={"ID":"b77c143f-8739-48c6-b37b-d6226bf36d0f","Type":"ContainerStarted","Data":"58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312"}
Nov 02 13:09:55.975246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:55.975230    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:09:56.021129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:56.021068    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl" podStartSLOduration=2.021051356 podStartE2EDuration="2.021051356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:56.016777525 +0000 UTC m=+2542.123480372" watchObservedRunningTime="2026-11-02 13:09:56.021051356 +0000 UTC m=+2542.127754198"
Nov 02 13:09:56.060484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:56.060417    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp" podStartSLOduration=2.060399538 podStartE2EDuration="2.060399538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:09:56.059948902 +0000 UTC m=+2542.166651747" watchObservedRunningTime="2026-11-02 13:09:56.060399538 +0000 UTC m=+2542.167102390"
Nov 02 13:09:56.679831 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:56.679797    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:56.827220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:56.827177    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t" podUID="ab1d0069-ae90-4f41-94f2-739840f6a71d" containerName="payload-processing" containerID="cri-o://3bd7faad6a045f85cf952b9056d5cb624bbc0c76dcd9d175dd973013b9f8768f" gracePeriod=25
Nov 02 13:09:56.988683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:56.988646    5879 generic.go:356] "Generic (PLEG): container finished" podID="ab1d0069-ae90-4f41-94f2-739840f6a71d" containerID="3bd7faad6a045f85cf952b9056d5cb624bbc0c76dcd9d175dd973013b9f8768f" exitCode=0
Nov 02 13:09:56.989153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:56.988721    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t" event={"ID":"ab1d0069-ae90-4f41-94f2-739840f6a71d","Type":"ContainerDied","Data":"3bd7faad6a045f85cf952b9056d5cb624bbc0c76dcd9d175dd973013b9f8768f"}
Nov 02 13:09:56.991133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:56.991094    5879 generic.go:356] "Generic (PLEG): container finished" podID="4704a4c4-6a62-4a71-90f2-cba9ede3dd5a" containerID="5cbe265c473543de5843a024cc0d6ffae7c01172204afdd2f39bea8a124b39f8" exitCode=0
Nov 02 13:09:56.991291 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:56.991174    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl" event={"ID":"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a","Type":"ContainerDied","Data":"5cbe265c473543de5843a024cc0d6ffae7c01172204afdd2f39bea8a124b39f8"}
Nov 02 13:09:57.123421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.123395    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:57.221782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.221691    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-kube-api-access-7zvng\" (UniqueName: \"kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-kube-api-access-7zvng\") pod \"ab1d0069-ae90-4f41-94f2-739840f6a71d\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") "
Nov 02 13:09:57.221947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.221869    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-routing-overlay-volume\") pod \"ab1d0069-ae90-4f41-94f2-739840f6a71d\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") "
Nov 02 13:09:57.221947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.221906    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-plugins-config-volume\") pod \"ab1d0069-ae90-4f41-94f2-739840f6a71d\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") "
Nov 02 13:09:57.222177 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.222160    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials\") pod \"ab1d0069-ae90-4f41-94f2-739840f6a71d\" (UID: \"ab1d0069-ae90-4f41-94f2-739840f6a71d\") "
Nov 02 13:09:57.222293 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.222268    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-routing-overlay-volume" pod "ab1d0069-ae90-4f41-94f2-739840f6a71d" (UID: "ab1d0069-ae90-4f41-94f2-739840f6a71d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:57.222504 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.222347    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-plugins-config-volume" pod "ab1d0069-ae90-4f41-94f2-739840f6a71d" (UID: "ab1d0069-ae90-4f41-94f2-739840f6a71d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:57.222853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.222833    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:57.222938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.222858    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab1d0069-ae90-4f41-94f2-739840f6a71d-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:57.224396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.224374    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-kube-api-access-7zvng" pod "ab1d0069-ae90-4f41-94f2-739840f6a71d" (UID: "ab1d0069-ae90-4f41-94f2-739840f6a71d"). InnerVolumeSpecName "kube-api-access-7zvng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:57.224714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.224698    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials" pod "ab1d0069-ae90-4f41-94f2-739840f6a71d" (UID: "ab1d0069-ae90-4f41-94f2-739840f6a71d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:57.325143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.325096    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:57.325143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:57.325137    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zvng\" (UniqueName: \"kubernetes.io/projected/ab1d0069-ae90-4f41-94f2-739840f6a71d-kube-api-access-7zvng\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:58.001892 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.001858    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"
Nov 02 13:09:58.002438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.001887    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t" event={"ID":"ab1d0069-ae90-4f41-94f2-739840f6a71d","Type":"ContainerDied","Data":"749a007ed7eae6ecfee49204ef264c48ee387b731c65f4c23ddce0384d3f2b79"}
Nov 02 13:09:58.002438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.001959    5879 scope.go:122] "RemoveContainer" containerID="3bd7faad6a045f85cf952b9056d5cb624bbc0c76dcd9d175dd973013b9f8768f"
Nov 02 13:09:58.065383 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.065292    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"]
Nov 02 13:09:58.086971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.080413    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b4d54cfdc-cph7t"]
Nov 02 13:09:58.177407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.177371    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:58.234377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.234332    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-xks-ca\") pod \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\" (UID: \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\") "
Nov 02 13:09:58.234597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.234418    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-openshift-ca\") pod \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\" (UID: \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\") "
Nov 02 13:09:58.234748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.234727    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-kube-api-access-9bv9z\" (UniqueName: \"kubernetes.io/projected/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-kube-api-access-9bv9z\") pod \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\" (UID: \"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a\") "
Nov 02 13:09:58.234843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.234825    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-openshift-ca" pod "4704a4c4-6a62-4a71-90f2-cba9ede3dd5a" (UID: "4704a4c4-6a62-4a71-90f2-cba9ede3dd5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:09:58.235399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.235367    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:58.237787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.237760    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-xks-ca" pod "4704a4c4-6a62-4a71-90f2-cba9ede3dd5a" (UID: "4704a4c4-6a62-4a71-90f2-cba9ede3dd5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:09:58.237901 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.237853    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-kube-api-access-9bv9z" pod "4704a4c4-6a62-4a71-90f2-cba9ede3dd5a" (UID: "4704a4c4-6a62-4a71-90f2-cba9ede3dd5a"). InnerVolumeSpecName "kube-api-access-9bv9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:09:58.336496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.336403    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bv9z\" (UniqueName: \"kubernetes.io/projected/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-kube-api-access-9bv9z\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:58.336496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.336433    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:09:58.625605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.625565    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:58.753337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.753304    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:58.890651 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.890561    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:09:58.951240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.951206    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"]
Nov 02 13:09:58.952052 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.952032    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ab1d0069-ae90-4f41-94f2-739840f6a71d" containerName="payload-processing"
Nov 02 13:09:58.952052 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.952049    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1d0069-ae90-4f41-94f2-739840f6a71d" containerName="payload-processing"
Nov 02 13:09:58.952252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.952062    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4704a4c4-6a62-4a71-90f2-cba9ede3dd5a" containerName="revoke-keys"
Nov 02 13:09:58.952252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.952067    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4704a4c4-6a62-4a71-90f2-cba9ede3dd5a" containerName="revoke-keys"
Nov 02 13:09:58.952252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.952212    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1d0069-ae90-4f41-94f2-739840f6a71d" containerName="payload-processing"
Nov 02 13:09:58.952252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.952221    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4704a4c4-6a62-4a71-90f2-cba9ede3dd5a" containerName="revoke-keys"
Nov 02 13:09:58.956437 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.956416    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"
Nov 02 13:09:58.973408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.973353    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-6a71db\"/\"default-dockercfg-zhmfl\""
Nov 02 13:09:58.973685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.973654    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-6a71db\"/\"e2e-usage-sim-233c03-cert\""
Nov 02 13:09:58.976566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.976544    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"]
Nov 02 13:09:58.985188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.985157    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-6a71db\"/\"kube-root-ca.crt\""
Nov 02 13:09:58.985343 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:58.985206    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-6a71db\"/\"openshift-service-ca.crt\""
Nov 02 13:09:59.015631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:59.015601    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"
Nov 02 13:09:59.015631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:59.015621    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl" event={"ID":"4704a4c4-6a62-4a71-90f2-cba9ede3dd5a","Type":"ContainerDied","Data":"1352d5d8ac8cbb9ebcfd199515bc1b63306f091fabdfcb6c52177b133093f60c"}
Nov 02 13:09:59.016085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:59.015668    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1352d5d8ac8cbb9ebcfd199515bc1b63306f091fabdfcb6c52177b133093f60c"
Nov 02 13:09:59.047837 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:59.047803    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgd5k\" (UniqueName: \"kubernetes.io/projected/f1c7693b-d870-4917-afc2-c27e80fe2b69-kube-api-access-xgd5k\") pod \"e2e-usage-sim-233c03-8c55d4647-bqz8q\" (UID: \"f1c7693b-d870-4917-afc2-c27e80fe2b69\") " pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"
Nov 02 13:09:59.048019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:59.047945    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f1c7693b-d870-4917-afc2-c27e80fe2b69-tls\") pod \"e2e-usage-sim-233c03-8c55d4647-bqz8q\" (UID: \"f1c7693b-d870-4917-afc2-c27e80fe2b69\") " pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"
Nov 02 13:09:59.149404 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:59.149289    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgd5k\" (UniqueName: \"kubernetes.io/projected/f1c7693b-d870-4917-afc2-c27e80fe2b69-kube-api-access-xgd5k\") pod \"e2e-usage-sim-233c03-8c55d4647-bqz8q\" (UID: \"f1c7693b-d870-4917-afc2-c27e80fe2b69\") " pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"
Nov 02 13:09:59.149593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:59.149441    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f1c7693b-d870-4917-afc2-c27e80fe2b69-tls\") pod \"e2e-usage-sim-233c03-8c55d4647-bqz8q\" (UID: \"f1c7693b-d870-4917-afc2-c27e80fe2b69\") " pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"
Nov 02 13:09:59.152123 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:59.152098    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f1c7693b-d870-4917-afc2-c27e80fe2b69-tls\") pod \"e2e-usage-sim-233c03-8c55d4647-bqz8q\" (UID: \"f1c7693b-d870-4917-afc2-c27e80fe2b69\") " pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"
Nov 02 13:09:59.175501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:59.175471    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgd5k\" (UniqueName: \"kubernetes.io/projected/f1c7693b-d870-4917-afc2-c27e80fe2b69-kube-api-access-xgd5k\") pod \"e2e-usage-sim-233c03-8c55d4647-bqz8q\" (UID: \"f1c7693b-d870-4917-afc2-c27e80fe2b69\") " pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"
Nov 02 13:09:59.271947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:59.271905    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"
Nov 02 13:09:59.579436 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:09:59.579389    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1c7693b_d870_4917_afc2_c27e80fe2b69.slice/crio-e8bb7d90adc8c0c7b2d24048c69e88bb47cdc3d4dd86d811aea872e92e0afad4 WatchSource:0}: Error finding container e8bb7d90adc8c0c7b2d24048c69e88bb47cdc3d4dd86d811aea872e92e0afad4: Status 404 returned error can't find the container with id e8bb7d90adc8c0c7b2d24048c69e88bb47cdc3d4dd86d811aea872e92e0afad4
Nov 02 13:09:59.581374 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:59.581339    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"]
Nov 02 13:09:59.886795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:09:59.886766    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:10:00.021929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.021897    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1d0069-ae90-4f41-94f2-739840f6a71d" path="/var/lib/kubelet/pods/ab1d0069-ae90-4f41-94f2-739840f6a71d/volumes"
Nov 02 13:10:00.029277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.029231    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q" event={"ID":"f1c7693b-d870-4917-afc2-c27e80fe2b69","Type":"ContainerStarted","Data":"cec7dbff10f8e1cc401c8e54723b26ac1a8f852d5ab32d09a0a9051c47a06c0e"}
Nov 02 13:10:00.029491 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.029287    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q" event={"ID":"f1c7693b-d870-4917-afc2-c27e80fe2b69","Type":"ContainerStarted","Data":"e8bb7d90adc8c0c7b2d24048c69e88bb47cdc3d4dd86d811aea872e92e0afad4"}
Nov 02 13:10:00.029491 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.029312    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"
Nov 02 13:10:00.034813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.034781    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"
Nov 02 13:10:00.036316 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.036296    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"]
Nov 02 13:10:00.041464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.041433    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a33c1a-5mmr5"]
Nov 02 13:10:00.075639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.075581    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q" podStartSLOduration=2.075565082 podStartE2EDuration="2.075565082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:10:00.072220417 +0000 UTC m=+2546.178923268" watchObservedRunningTime="2026-11-02 13:10:00.075565082 +0000 UTC m=+2546.182267969"
Nov 02 13:10:00.199611 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.199523    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"]
Nov 02 13:10:00.204749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.204723    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.223360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.223329    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-25f7a6-openshift-default-dockercfg-z957v\""
Nov 02 13:10:00.236607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.236519    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"]
Nov 02 13:10:00.267422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.267382    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-envoy\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.267619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.267431    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-data\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.267619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.267600    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-podinfo\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.267810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.267789    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-token\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.267927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.267909    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-certs\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.267978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.267957    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-credential-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.268111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.268094    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.268220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.268205    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xst8x\" (UniqueName: \"kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-kube-api-access-xst8x\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.268298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.268280    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istiod-ca-cert\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.369802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.369754    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istiod-ca-cert\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.369842    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-envoy\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.369867    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-data\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.369930    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-podinfo\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.370146    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-token\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.370244    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-certs\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.370284    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-credential-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.370326    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-data\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.370341    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370504 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.370473    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xst8x\" (UniqueName: \"kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-kube-api-access-xst8x\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.370591    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-certs\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.370627    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istiod-ca-cert\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.370629    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-credential-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.370789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.370691    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-socket\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.372823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.372797    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-envoy\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.372946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.372879    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-podinfo\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.396171 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.396135    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xst8x\" (UniqueName: \"kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-kube-api-access-xst8x\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.396745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.396717    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-token\") pod \"e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") " pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.494901 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.494803    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:00.527425 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.527376    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:00.857196 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:10:00.857165    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8847cdca_ae49_47a9_8dcc_a7e655fc2ad7.slice/crio-45efd20996a533d8112b021a4f229b93b7b38c93ee2b266efc4eee85793abcd3 WatchSource:0}: Error finding container 45efd20996a533d8112b021a4f229b93b7b38c93ee2b266efc4eee85793abcd3: Status 404 returned error can't find the container with id 45efd20996a533d8112b021a4f229b93b7b38c93ee2b266efc4eee85793abcd3
Nov 02 13:10:00.860008 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.859972    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"]
Nov 02 13:10:00.860605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.860572    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:10:00.860653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.860641    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:10:00.860686 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:00.860675    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:10:01.041818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:01.041776    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh" event={"ID":"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7","Type":"ContainerStarted","Data":"689eca75a6c64c18fe151f7e0f4dc4792947d20b537acffdd7a76d8ea365708a"}
Nov 02 13:10:01.043346 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:01.041841    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh" event={"ID":"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7","Type":"ContainerStarted","Data":"45efd20996a533d8112b021a4f229b93b7b38c93ee2b266efc4eee85793abcd3"}
Nov 02 13:10:01.091309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:01.091251    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh" podStartSLOduration=1.091235337 podStartE2EDuration="1.091235337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:10:01.088608846 +0000 UTC m=+2547.195311708" watchObservedRunningTime="2026-11-02 13:10:01.091235337 +0000 UTC m=+2547.197938188"
Nov 02 13:10:01.528681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:01.528639    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:01.530258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:01.530232    5879 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Nov 02 13:10:01.530368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:01.530284    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh" podUID="8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Nov 02 13:10:01.913428 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:01.913397    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:10:01.913849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:01.913826    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:10:02.031407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:02.027855    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"]
Nov 02 13:10:02.046630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:02.046590    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c07b02b-5bd3-43ac-864b-eb5b5606c05d" path="/var/lib/kubelet/pods/5c07b02b-5bd3-43ac-864b-eb5b5606c05d/volumes"
Nov 02 13:10:02.049745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:02.048973    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"]
Nov 02 13:10:02.422177 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:02.422139    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:02.527977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:02.527943    5879 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Nov 02 13:10:02.528177 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:02.528041    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh" podUID="8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Nov 02 13:10:03.528518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:03.528387    5879 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Nov 02 13:10:03.528518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:03.528440    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh" podUID="8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Nov 02 13:10:04.307089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:04.306982    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:04.527862 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:04.527825    5879 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Nov 02 13:10:04.528060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:04.527875    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh" podUID="8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Nov 02 13:10:04.915023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:04.912722    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"]
Nov 02 13:10:04.935831 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:04.935747    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:04.948013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:04.938374    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:04.956372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:04.956345    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:10:04.962614 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:04.962579    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"]
Nov 02 13:10:04.973681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:04.973639    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:10:05.060057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.059501    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-512c850a-7ll95\" (UID: \"e0515ea2-a0ef-408b-a552-025591ee29de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:05.060057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.059715    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-512c850a-7ll95\" (UID: \"e0515ea2-a0ef-408b-a552-025591ee29de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:05.060057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.059808    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6c85\" (UniqueName: \"kubernetes.io/projected/e0515ea2-a0ef-408b-a552-025591ee29de-kube-api-access-r6c85\") pod \"maas-api-revoke-sub-512c850a-7ll95\" (UID: \"e0515ea2-a0ef-408b-a552-025591ee29de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:05.067965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.067931    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:05.163015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.162530    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-512c850a-7ll95\" (UID: \"e0515ea2-a0ef-408b-a552-025591ee29de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:05.163015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.162704    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6c85\" (UniqueName: \"kubernetes.io/projected/e0515ea2-a0ef-408b-a552-025591ee29de-kube-api-access-r6c85\") pod \"maas-api-revoke-sub-512c850a-7ll95\" (UID: \"e0515ea2-a0ef-408b-a552-025591ee29de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:05.163015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.162805    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-512c850a-7ll95\" (UID: \"e0515ea2-a0ef-408b-a552-025591ee29de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:05.166073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.163771    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-512c850a-7ll95\" (UID: \"e0515ea2-a0ef-408b-a552-025591ee29de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:05.178871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.169243    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-512c850a-7ll95\" (UID: \"e0515ea2-a0ef-408b-a552-025591ee29de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:05.219424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.209211    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6c85\" (UniqueName: \"kubernetes.io/projected/e0515ea2-a0ef-408b-a552-025591ee29de-kube-api-access-r6c85\") pod \"maas-api-revoke-sub-512c850a-7ll95\" (UID: \"e0515ea2-a0ef-408b-a552-025591ee29de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:05.268211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.268179    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:05.528513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.528416    5879 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Nov 02 13:10:05.528797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:05.528768    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh" podUID="8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Nov 02 13:10:06.031289 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:10:06.031252    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0515ea2_a0ef_408b_a552_025591ee29de.slice/crio-1575924a062e94d851bfcbe4ab87fe86b81b4991f67b66fa5bf4a4c4d6a1deb4 WatchSource:0}: Error finding container 1575924a062e94d851bfcbe4ab87fe86b81b4991f67b66fa5bf4a4c4d6a1deb4: Status 404 returned error can't find the container with id 1575924a062e94d851bfcbe4ab87fe86b81b4991f67b66fa5bf4a4c4d6a1deb4
Nov 02 13:10:06.038589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.038488    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:10:06.038589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.038544    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"]
Nov 02 13:10:06.146540 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.146506    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95" event={"ID":"e0515ea2-a0ef-408b-a552-025591ee29de","Type":"ContainerStarted","Data":"1575924a062e94d851bfcbe4ab87fe86b81b4991f67b66fa5bf4a4c4d6a1deb4"}
Nov 02 13:10:06.248161 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.248128    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"]
Nov 02 13:10:06.254774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.254749    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.276782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.276753    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-95ee26-openshift-default-dockercfg-xjh8d\""
Nov 02 13:10:06.278092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.278021    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"]
Nov 02 13:10:06.444237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.444195    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.444438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.444255    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.444438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.444391    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.444533 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.444464    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.444533 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.444526    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttnvb\" (UniqueName: \"kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-kube-api-access-ttnvb\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.444624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.444570    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.444624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.444602    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.444700 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.444656    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.444700 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.444686    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262814fe-6da4-48ae-85b8-c51aa9f2895a-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.528497 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.528416    5879 patch_prober.go:29] interesting pod/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body=
Nov 02 13:10:06.528497 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.528476    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh" podUID="8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused"
Nov 02 13:10:06.546101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.546050    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.546249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.546128    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.546289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.546254    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.546366 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.546350    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.546469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.546453    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnvb\" (UniqueName: \"kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-kube-api-access-ttnvb\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.546507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.546489    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.546545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.546532    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.546593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.546578    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.546639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.546625    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262814fe-6da4-48ae-85b8-c51aa9f2895a-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.546724 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.546707    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-data\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.546814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.546775    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.546852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.546811    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-certs\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.547127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.547102    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-credential-socket\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.547377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.547358    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262814fe-6da4-48ae-85b8-c51aa9f2895a-istiod-ca-cert\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.549590 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.549571    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-podinfo\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.550109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.550086    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-envoy\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.572899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.572860    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-token\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.573384 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.573348    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnvb\" (UniqueName: \"kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-kube-api-access-ttnvb\") pod \"e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") " pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.872716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.872676    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:06.908592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:06.908505    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:07.022528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.015883    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:10:07.030634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.030594    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr" podUID="dcc6bf3f-d673-4095-bf6c-8a81b66fe351" containerName="payload-processing" containerID="cri-o://d9c6509e1b5d56e145628f8cd37641a6f865e8e5d111fe871bc7605e2c5e0869" gracePeriod=25
Nov 02 13:10:07.050250 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.050199    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf" podUID="4c6522c4-39f2-4057-9e66-594e92c504a3" containerName="payload-processing" containerID="cri-o://b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163" gracePeriod=25
Nov 02 13:10:07.074443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.060823    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"]
Nov 02 13:10:07.120488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.120455    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.139070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.138572    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 13:10:07.141281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.139074    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 13:10:07.141281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.139299    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6fgxx\""
Nov 02 13:10:07.141281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.139415    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 13:10:07.147824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.147788    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"]
Nov 02 13:10:07.156035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.155204    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"]
Nov 02 13:10:07.170074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.168372    5879 generic.go:356] "Generic (PLEG): container finished" podID="dcc6bf3f-d673-4095-bf6c-8a81b66fe351" containerID="d9c6509e1b5d56e145628f8cd37641a6f865e8e5d111fe871bc7605e2c5e0869" exitCode=0
Nov 02 13:10:07.170074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.168495    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr" event={"ID":"dcc6bf3f-d673-4095-bf6c-8a81b66fe351","Type":"ContainerDied","Data":"d9c6509e1b5d56e145628f8cd37641a6f865e8e5d111fe871bc7605e2c5e0869"}
Nov 02 13:10:07.210236 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.209110    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95" event={"ID":"e0515ea2-a0ef-408b-a552-025591ee29de","Type":"ContainerStarted","Data":"e95b8bca830d4b72d0c3338d1c3c380bcff7ffceefd9ce6fcd58baae980e8771"}
Nov 02 13:10:07.267472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.265806    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.267472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.265865    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cftm\" (UniqueName: \"kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-kube-api-access-2cftm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.267472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.265916    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.267472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.266080    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.267472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.266284    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.267472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.266390    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.267472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.266483    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.316440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.316392    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:10:07.316683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.316657    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:10:07.316828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.316809    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:10:07.319172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.319151    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"]
Nov 02 13:10:07.326482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.326427    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95" podStartSLOduration=3.326408077 podStartE2EDuration="3.326408077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:10:07.30775832 +0000 UTC m=+2553.414461171" watchObservedRunningTime="2026-11-02 13:10:07.326408077 +0000 UTC m=+2553.433110929"
Nov 02 13:10:07.387061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.378109    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.387061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.378206    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.387061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.378265    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cftm\" (UniqueName: \"kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-kube-api-access-2cftm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.387061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.378351    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.387061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.378538    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.387061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.378588    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.387061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.378676    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.393402 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.390271    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.399028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.396109    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.399028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.396719    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.399028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.399017    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.400756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.400727    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.415439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.415398    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cftm\" (UniqueName: \"kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-kube-api-access-2cftm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.417446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.417402    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wr8nm\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.497771 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.497393    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:10:07.502311 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.502284    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:10:07.502578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.502563    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:07.548461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.548208    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:07.550309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.550282    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:07.557169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.557134    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:10:07.684634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.684557    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-plugins-config-volume\") pod \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") "
Nov 02 13:10:07.684634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.684605    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-routing-overlay-volume\") pod \"4c6522c4-39f2-4057-9e66-594e92c504a3\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") "
Nov 02 13:10:07.684825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.684639    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-provider-credentials\") pod \"4c6522c4-39f2-4057-9e66-594e92c504a3\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") "
Nov 02 13:10:07.684825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.684656    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-kube-api-access-58hn6\" (UniqueName: \"kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-kube-api-access-58hn6\") pod \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") "
Nov 02 13:10:07.684825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.684708    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-plugins-config-volume\") pod \"4c6522c4-39f2-4057-9e66-594e92c504a3\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") "
Nov 02 13:10:07.684825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.684816    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-kube-api-access-jxlb8\" (UniqueName: \"kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-kube-api-access-jxlb8\") pod \"4c6522c4-39f2-4057-9e66-594e92c504a3\" (UID: \"4c6522c4-39f2-4057-9e66-594e92c504a3\") "
Nov 02 13:10:07.684956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.684923    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-routing-overlay-volume\") pod \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") "
Nov 02 13:10:07.684956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.684941    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-provider-credentials\") pod \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\" (UID: \"dcc6bf3f-d673-4095-bf6c-8a81b66fe351\") "
Nov 02 13:10:07.685936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.685907    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-plugins-config-volume" pod "4c6522c4-39f2-4057-9e66-594e92c504a3" (UID: "4c6522c4-39f2-4057-9e66-594e92c504a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:07.686511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.686301    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-routing-overlay-volume" pod "dcc6bf3f-d673-4095-bf6c-8a81b66fe351" (UID: "dcc6bf3f-d673-4095-bf6c-8a81b66fe351"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:07.686511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.686461    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-plugins-config-volume" pod "dcc6bf3f-d673-4095-bf6c-8a81b66fe351" (UID: "dcc6bf3f-d673-4095-bf6c-8a81b66fe351"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:07.686691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.686657    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-routing-overlay-volume" pod "4c6522c4-39f2-4057-9e66-594e92c504a3" (UID: "4c6522c4-39f2-4057-9e66-594e92c504a3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:07.688788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.688736    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-provider-credentials" pod "dcc6bf3f-d673-4095-bf6c-8a81b66fe351" (UID: "dcc6bf3f-d673-4095-bf6c-8a81b66fe351"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:07.688788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.688774    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-kube-api-access-58hn6" pod "dcc6bf3f-d673-4095-bf6c-8a81b66fe351" (UID: "dcc6bf3f-d673-4095-bf6c-8a81b66fe351"). InnerVolumeSpecName "kube-api-access-58hn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:07.689258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.689238    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-kube-api-access-jxlb8" pod "4c6522c4-39f2-4057-9e66-594e92c504a3" (UID: "4c6522c4-39f2-4057-9e66-594e92c504a3"). InnerVolumeSpecName "kube-api-access-jxlb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:07.690124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.690106    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-provider-credentials" pod "4c6522c4-39f2-4057-9e66-594e92c504a3" (UID: "4c6522c4-39f2-4057-9e66-594e92c504a3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:07.787461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.787416    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:07.787461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.787461    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:07.787750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.787477    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58hn6\" (UniqueName: \"kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-kube-api-access-58hn6\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:07.787750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.787491    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c6522c4-39f2-4057-9e66-594e92c504a3-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:07.787750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.787504    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxlb8\" (UniqueName: \"kubernetes.io/projected/4c6522c4-39f2-4057-9e66-594e92c504a3-kube-api-access-jxlb8\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:07.787750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.787518    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:07.787750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.787531    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:07.787750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.787545    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcc6bf3f-d673-4095-bf6c-8a81b66fe351-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:07.880701 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:10:07.880666    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1afc3d86_7a0c_4f97_9298_204bd8bb0d2b.slice/crio-5301946cb76e3fbd216aa3a20b1d5ab72afe138b52a172f96134c5ffbfd350b0 WatchSource:0}: Error finding container 5301946cb76e3fbd216aa3a20b1d5ab72afe138b52a172f96134c5ffbfd350b0: Status 404 returned error can't find the container with id 5301946cb76e3fbd216aa3a20b1d5ab72afe138b52a172f96134c5ffbfd350b0
Nov 02 13:10:07.884322 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.884276    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:10:07.884430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.884371    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:10:07.884527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:07.884507    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"]
Nov 02 13:10:08.227812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.227775    5879 generic.go:356] "Generic (PLEG): container finished" podID="4c6522c4-39f2-4057-9e66-594e92c504a3" containerID="b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163" exitCode=0
Nov 02 13:10:08.227812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.227825    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"
Nov 02 13:10:08.228399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.227854    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf" event={"ID":"4c6522c4-39f2-4057-9e66-594e92c504a3","Type":"ContainerDied","Data":"b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163"}
Nov 02 13:10:08.228399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.227917    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf" event={"ID":"4c6522c4-39f2-4057-9e66-594e92c504a3","Type":"ContainerDied","Data":"465172e8ca361ce24c590ae726920b329e7ce0fcaf05d1f8538b9090a6ba40c1"}
Nov 02 13:10:08.228399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.227944    5879 scope.go:122] "RemoveContainer" containerID="b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163"
Nov 02 13:10:08.230881 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.230857    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm" event={"ID":"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b","Type":"ContainerStarted","Data":"6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b"}
Nov 02 13:10:08.231064 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.230902    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm" event={"ID":"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b","Type":"ContainerStarted","Data":"5301946cb76e3fbd216aa3a20b1d5ab72afe138b52a172f96134c5ffbfd350b0"}
Nov 02 13:10:08.231247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.231229    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:08.232588 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.232568    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"
Nov 02 13:10:08.232710 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.232602    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr" event={"ID":"dcc6bf3f-d673-4095-bf6c-8a81b66fe351","Type":"ContainerDied","Data":"643f64f30ca6cfd19c91ea49de439e839ec523c524b20de9d5cb359e101615ab"}
Nov 02 13:10:08.233447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.233424    5879 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wr8nm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:10:08.233559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.233489    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm" podUID="1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:10:08.235624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.235537    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9" event={"ID":"262814fe-6da4-48ae-85b8-c51aa9f2895a","Type":"ContainerStarted","Data":"82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941"}
Nov 02 13:10:08.235624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.235591    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9" event={"ID":"262814fe-6da4-48ae-85b8-c51aa9f2895a","Type":"ContainerStarted","Data":"44edf89609c365a31dce6a24a342b4f61751be8f06ad1ff89286bd32ac23bff9"}
Nov 02 13:10:08.237536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.237500    5879 generic.go:356] "Generic (PLEG): container finished" podID="e0515ea2-a0ef-408b-a552-025591ee29de" containerID="e95b8bca830d4b72d0c3338d1c3c380bcff7ffceefd9ce6fcd58baae980e8771" exitCode=0
Nov 02 13:10:08.237644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.237595    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95" event={"ID":"e0515ea2-a0ef-408b-a552-025591ee29de","Type":"ContainerDied","Data":"e95b8bca830d4b72d0c3338d1c3c380bcff7ffceefd9ce6fcd58baae980e8771"}
Nov 02 13:10:08.242789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.242711    5879 scope.go:122] "RemoveContainer" containerID="b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163"
Nov 02 13:10:08.243070 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:10:08.243052    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163\": container with ID starting with b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163 not found: ID does not exist" containerID="b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163"
Nov 02 13:10:08.243148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.243081    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163"} err="failed to get container status \"b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163\": rpc error: code = NotFound desc = could not find container \"b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163\": container with ID starting with b8d691be6dad021c84a67947a3e082af3215e44e3e92b5391aca3f9c0ff3b163 not found: ID does not exist"
Nov 02 13:10:08.243148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.243102    5879 scope.go:122] "RemoveContainer" containerID="d9c6509e1b5d56e145628f8cd37641a6f865e8e5d111fe871bc7605e2c5e0869"
Nov 02 13:10:08.304020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.303936    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9" podStartSLOduration=2.303918046 podStartE2EDuration="2.303918046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:10:08.302876312 +0000 UTC m=+2554.409579166" watchObservedRunningTime="2026-11-02 13:10:08.303918046 +0000 UTC m=+2554.410620900"
Nov 02 13:10:08.346468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.346437    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"]
Nov 02 13:10:08.353243 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.353211    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-64c5lc6pf"]
Nov 02 13:10:08.416073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.416038    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"]
Nov 02 13:10:08.422678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.422643    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-5559cftzr"]
Nov 02 13:10:08.533151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.533025    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm" podStartSLOduration=1.532982544 podStartE2EDuration="1.532982544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:10:08.488530332 +0000 UTC m=+2554.595233191" watchObservedRunningTime="2026-11-02 13:10:08.532982544 +0000 UTC m=+2554.639685395"
Nov 02 13:10:08.873297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.873264    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:08.875103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.875071    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Nov 02 13:10:08.875261 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:08.875129    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9" podUID="262814fe-6da4-48ae-85b8-c51aa9f2895a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Nov 02 13:10:09.007344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.007258    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:09.129477 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.129400    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:09.251779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.251749    5879 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wr8nm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:10:09.252153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.251808    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm" podUID="1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:10:09.533296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.533258    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:09.565105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.565069    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-openshift-ca\") pod \"e0515ea2-a0ef-408b-a552-025591ee29de\" (UID: \"e0515ea2-a0ef-408b-a552-025591ee29de\") "
Nov 02 13:10:09.565303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.565179    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-xks-ca\") pod \"e0515ea2-a0ef-408b-a552-025591ee29de\" (UID: \"e0515ea2-a0ef-408b-a552-025591ee29de\") "
Nov 02 13:10:09.565438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.565413    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0515ea2-a0ef-408b-a552-025591ee29de-kube-api-access-r6c85\" (UniqueName: \"kubernetes.io/projected/e0515ea2-a0ef-408b-a552-025591ee29de-kube-api-access-r6c85\") pod \"e0515ea2-a0ef-408b-a552-025591ee29de\" (UID: \"e0515ea2-a0ef-408b-a552-025591ee29de\") "
Nov 02 13:10:09.565606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.565516    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-openshift-ca" pod "e0515ea2-a0ef-408b-a552-025591ee29de" (UID: "e0515ea2-a0ef-408b-a552-025591ee29de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:09.566134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.566072    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:09.568476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.568448    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-xks-ca" pod "e0515ea2-a0ef-408b-a552-025591ee29de" (UID: "e0515ea2-a0ef-408b-a552-025591ee29de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:10:09.568587 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.568484    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0515ea2-a0ef-408b-a552-025591ee29de-kube-api-access-r6c85" pod "e0515ea2-a0ef-408b-a552-025591ee29de" (UID: "e0515ea2-a0ef-408b-a552-025591ee29de"). InnerVolumeSpecName "kube-api-access-r6c85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:09.667110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.667072    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6c85\" (UniqueName: \"kubernetes.io/projected/e0515ea2-a0ef-408b-a552-025591ee29de-kube-api-access-r6c85\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:09.667338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.667325    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0515ea2-a0ef-408b-a552-025591ee29de-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:09.874169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.874137    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Nov 02 13:10:09.874320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:09.874190    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9" podUID="262814fe-6da4-48ae-85b8-c51aa9f2895a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Nov 02 13:10:10.019178 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:10.019139    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6522c4-39f2-4057-9e66-594e92c504a3" path="/var/lib/kubelet/pods/4c6522c4-39f2-4057-9e66-594e92c504a3/volumes"
Nov 02 13:10:10.019846 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:10.019824    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc6bf3f-d673-4095-bf6c-8a81b66fe351" path="/var/lib/kubelet/pods/dcc6bf3f-d673-4095-bf6c-8a81b66fe351/volumes"
Nov 02 13:10:10.260413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:10.260327    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"
Nov 02 13:10:10.260413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:10.260336    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95" event={"ID":"e0515ea2-a0ef-408b-a552-025591ee29de","Type":"ContainerDied","Data":"1575924a062e94d851bfcbe4ab87fe86b81b4991f67b66fa5bf4a4c4d6a1deb4"}
Nov 02 13:10:10.260413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:10.260391    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1575924a062e94d851bfcbe4ab87fe86b81b4991f67b66fa5bf4a4c4d6a1deb4"
Nov 02 13:10:10.851745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:10.851711    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:10.874049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:10.874011    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Nov 02 13:10:10.874337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:10.874073    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9" podUID="262814fe-6da4-48ae-85b8-c51aa9f2895a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Nov 02 13:10:11.873193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:11.873158    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Nov 02 13:10:11.873619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:11.873213    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9" podUID="262814fe-6da4-48ae-85b8-c51aa9f2895a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Nov 02 13:10:12.156649 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.156560    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l" podUID="9123562e-d96a-4a86-9764-519429f8f5f6" containerName="payload-processing" containerID="cri-o://828b5e48463f0cf3f1df8b79f51b8c2acac45558a2f8d277012edc02966f5da8" gracePeriod=25
Nov 02 13:10:12.251726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.251694    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:10:12.280346 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.280316    5879 generic.go:356] "Generic (PLEG): container finished" podID="9123562e-d96a-4a86-9764-519429f8f5f6" containerID="828b5e48463f0cf3f1df8b79f51b8c2acac45558a2f8d277012edc02966f5da8" exitCode=0
Nov 02 13:10:12.280510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.280390    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l" event={"ID":"9123562e-d96a-4a86-9764-519429f8f5f6","Type":"ContainerDied","Data":"828b5e48463f0cf3f1df8b79f51b8c2acac45558a2f8d277012edc02966f5da8"}
Nov 02 13:10:12.564317 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.564288    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:10:12.640339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.640306    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-kube-api-access-2pzvh\" (UniqueName: \"kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-kube-api-access-2pzvh\") pod \"9123562e-d96a-4a86-9764-519429f8f5f6\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") "
Nov 02 13:10:12.640515 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.640389    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-routing-overlay-volume\") pod \"9123562e-d96a-4a86-9764-519429f8f5f6\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") "
Nov 02 13:10:12.640515 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.640442    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-provider-credentials\") pod \"9123562e-d96a-4a86-9764-519429f8f5f6\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") "
Nov 02 13:10:12.640594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.640523    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-plugins-config-volume\") pod \"9123562e-d96a-4a86-9764-519429f8f5f6\" (UID: \"9123562e-d96a-4a86-9764-519429f8f5f6\") "
Nov 02 13:10:12.641618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.641594    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-routing-overlay-volume" pod "9123562e-d96a-4a86-9764-519429f8f5f6" (UID: "9123562e-d96a-4a86-9764-519429f8f5f6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:12.641793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.641767    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-plugins-config-volume" pod "9123562e-d96a-4a86-9764-519429f8f5f6" (UID: "9123562e-d96a-4a86-9764-519429f8f5f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:12.646531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.646504    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-kube-api-access-2pzvh" pod "9123562e-d96a-4a86-9764-519429f8f5f6" (UID: "9123562e-d96a-4a86-9764-519429f8f5f6"). InnerVolumeSpecName "kube-api-access-2pzvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:12.648434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.648407    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-provider-credentials" pod "9123562e-d96a-4a86-9764-519429f8f5f6" (UID: "9123562e-d96a-4a86-9764-519429f8f5f6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:12.742447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.742410    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pzvh\" (UniqueName: \"kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-kube-api-access-2pzvh\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:12.742447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.742442    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:12.742447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.742452    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9123562e-d96a-4a86-9764-519429f8f5f6-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:12.742708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.742467    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9123562e-d96a-4a86-9764-519429f8f5f6-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:12.827830 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.827740    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:12.873925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.873894    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Nov 02 13:10:12.874328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:12.873948    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9" podUID="262814fe-6da4-48ae-85b8-c51aa9f2895a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Nov 02 13:10:13.315498 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:13.315467    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"
Nov 02 13:10:13.315498 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:13.315490    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l" event={"ID":"9123562e-d96a-4a86-9764-519429f8f5f6","Type":"ContainerDied","Data":"bfcb31c112635e680286d3485a32110c03c3e82ba496a960bf21c9bfb162eff1"}
Nov 02 13:10:13.315786 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:13.315575    5879 scope.go:122] "RemoveContainer" containerID="828b5e48463f0cf3f1df8b79f51b8c2acac45558a2f8d277012edc02966f5da8"
Nov 02 13:10:13.374641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:13.374601    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"]
Nov 02 13:10:13.413022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:13.412403    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-5c97zk26l"]
Nov 02 13:10:13.873445 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:13.873403    5879 patch_prober.go:29] interesting pod/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body=
Nov 02 13:10:13.873635 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:13.873475    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9" podUID="262814fe-6da4-48ae-85b8-c51aa9f2895a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused"
Nov 02 13:10:14.020194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:14.020155    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9123562e-d96a-4a86-9764-519429f8f5f6" path="/var/lib/kubelet/pods/9123562e-d96a-4a86-9764-519429f8f5f6/volumes"
Nov 02 13:10:14.657150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:14.657113    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:14.823706 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:10:14.823666    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:10:14.998118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:14.997986    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:14.999708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:14.999683    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:15.001716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:15.001687    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:10:16.432014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:16.431971    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:16.558920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:16.558871    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:17.491028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:17.490968    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"]
Nov 02 13:10:17.513553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:17.513497    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:17.642153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:17.642116    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:18.140332 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:18.140295    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:18.504331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:18.504234    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:20.270298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:20.270265    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:20.327610 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:20.327508    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"]
Nov 02 13:10:21.865626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:21.865579    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:22.483829 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:22.483615    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:22.495865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:22.495071    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp" podUID="b77c143f-8739-48c6-b37b-d6226bf36d0f" containerName="payload-processing" containerID="cri-o://58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312" gracePeriod=25
Nov 02 13:10:23.265165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.265069    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:10:23.371976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.370967    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-provider-credentials\") pod \"b77c143f-8739-48c6-b37b-d6226bf36d0f\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") "
Nov 02 13:10:23.371976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.371082    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-kube-api-access-82vpc\" (UniqueName: \"kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-kube-api-access-82vpc\") pod \"b77c143f-8739-48c6-b37b-d6226bf36d0f\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") "
Nov 02 13:10:23.371976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.371126    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-routing-overlay-volume\") pod \"b77c143f-8739-48c6-b37b-d6226bf36d0f\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") "
Nov 02 13:10:23.371976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.371180    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-plugins-config-volume\") pod \"b77c143f-8739-48c6-b37b-d6226bf36d0f\" (UID: \"b77c143f-8739-48c6-b37b-d6226bf36d0f\") "
Nov 02 13:10:23.372397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.372246    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-routing-overlay-volume" pod "b77c143f-8739-48c6-b37b-d6226bf36d0f" (UID: "b77c143f-8739-48c6-b37b-d6226bf36d0f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:23.373065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.373041    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-plugins-config-volume" pod "b77c143f-8739-48c6-b37b-d6226bf36d0f" (UID: "b77c143f-8739-48c6-b37b-d6226bf36d0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:23.376757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.376730    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-kube-api-access-82vpc" pod "b77c143f-8739-48c6-b37b-d6226bf36d0f" (UID: "b77c143f-8739-48c6-b37b-d6226bf36d0f"). InnerVolumeSpecName "kube-api-access-82vpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:23.376928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.376816    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-provider-credentials" pod "b77c143f-8739-48c6-b37b-d6226bf36d0f" (UID: "b77c143f-8739-48c6-b37b-d6226bf36d0f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:23.466030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.465623    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-27twq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:10:23.466030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.465699    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podUID="c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:10:23.479783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.479744    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:23.480061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.480040    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82vpc\" (UniqueName: \"kubernetes.io/projected/b77c143f-8739-48c6-b37b-d6226bf36d0f-kube-api-access-82vpc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:23.480193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.480180    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:23.480294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.480282    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b77c143f-8739-48c6-b37b-d6226bf36d0f-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:23.529029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.528906    5879 generic.go:356] "Generic (PLEG): container finished" podID="b77c143f-8739-48c6-b37b-d6226bf36d0f" containerID="58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312" exitCode=0
Nov 02 13:10:23.529766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.529448    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"
Nov 02 13:10:23.533041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.529472    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp" event={"ID":"b77c143f-8739-48c6-b37b-d6226bf36d0f","Type":"ContainerDied","Data":"58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312"}
Nov 02 13:10:23.533282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.533257    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp" event={"ID":"b77c143f-8739-48c6-b37b-d6226bf36d0f","Type":"ContainerDied","Data":"3a84c4a9078743c8f93eb906d12f35da83a9e90f28e41ca7e796febf23011a1d"}
Nov 02 13:10:23.539022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.533417    5879 scope.go:122] "RemoveContainer" containerID="58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312"
Nov 02 13:10:23.616244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.615469    5879 scope.go:122] "RemoveContainer" containerID="58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312"
Nov 02 13:10:23.616244 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:10:23.615915    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312\": container with ID starting with 58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312 not found: ID does not exist" containerID="58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312"
Nov 02 13:10:23.616244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.615950    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312"} err="failed to get container status \"58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312\": rpc error: code = NotFound desc = could not find container \"58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312\": container with ID starting with 58fd08d3086e3f8a7e7e03a362a013dd0c3a16579d4cd100823b6b040f880312 not found: ID does not exist"
Nov 02 13:10:23.618468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.617182    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sg8t7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:10:23.618468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.617257    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:10:23.672016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.671161    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"]
Nov 02 13:10:23.696016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:23.690266    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-ef5efe/payload-processing-external-model-e2e-extmod-a-ef5efe-6bf9z5mfp"]
Nov 02 13:10:24.019793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:24.019764    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b77c143f-8739-48c6-b37b-d6226bf36d0f" path="/var/lib/kubelet/pods/b77c143f-8739-48c6-b37b-d6226bf36d0f/volumes"
Nov 02 13:10:24.472663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:24.472625    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:24.755620 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:24.755495    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:24.893425 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:24.893366    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:25.022080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.021953    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:25.328299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.328192    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj" podUID="7c066765-c46e-42e1-8d73-75bddd1dce76" containerName="payload-processing" containerID="cri-o://69e4ee0c3c44fb26bb164956b18a208280d4ee1ebbc768b5e7034667e3e21af3" gracePeriod=25
Nov 02 13:10:25.557580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.557547    5879 generic.go:356] "Generic (PLEG): container finished" podID="7c066765-c46e-42e1-8d73-75bddd1dce76" containerID="69e4ee0c3c44fb26bb164956b18a208280d4ee1ebbc768b5e7034667e3e21af3" exitCode=0
Nov 02 13:10:25.557938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.557629    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj" event={"ID":"7c066765-c46e-42e1-8d73-75bddd1dce76","Type":"ContainerDied","Data":"69e4ee0c3c44fb26bb164956b18a208280d4ee1ebbc768b5e7034667e3e21af3"}
Nov 02 13:10:25.622550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.622522    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:10:25.724461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.724409    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-plugins-config-volume\") pod \"7c066765-c46e-42e1-8d73-75bddd1dce76\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") "
Nov 02 13:10:25.724653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.724511    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-provider-credentials\") pod \"7c066765-c46e-42e1-8d73-75bddd1dce76\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") "
Nov 02 13:10:25.724709 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.724648    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-kube-api-access-26x54\" (UniqueName: \"kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-kube-api-access-26x54\") pod \"7c066765-c46e-42e1-8d73-75bddd1dce76\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") "
Nov 02 13:10:25.724757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.724722    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-routing-overlay-volume\") pod \"7c066765-c46e-42e1-8d73-75bddd1dce76\" (UID: \"7c066765-c46e-42e1-8d73-75bddd1dce76\") "
Nov 02 13:10:25.724869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.724841    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-plugins-config-volume" pod "7c066765-c46e-42e1-8d73-75bddd1dce76" (UID: "7c066765-c46e-42e1-8d73-75bddd1dce76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:25.725487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.725457    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-routing-overlay-volume" pod "7c066765-c46e-42e1-8d73-75bddd1dce76" (UID: "7c066765-c46e-42e1-8d73-75bddd1dce76"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:25.725670 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.725646    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:25.725811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.725673    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c066765-c46e-42e1-8d73-75bddd1dce76-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:25.727107 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.727083    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-kube-api-access-26x54" pod "7c066765-c46e-42e1-8d73-75bddd1dce76" (UID: "7c066765-c46e-42e1-8d73-75bddd1dce76"). InnerVolumeSpecName "kube-api-access-26x54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:25.727220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.727108    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-provider-credentials" pod "7c066765-c46e-42e1-8d73-75bddd1dce76" (UID: "7c066765-c46e-42e1-8d73-75bddd1dce76"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:25.826667 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.826633    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:25.826667 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:25.826663    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26x54\" (UniqueName: \"kubernetes.io/projected/7c066765-c46e-42e1-8d73-75bddd1dce76-kube-api-access-26x54\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:26.326445 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:26.326361    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:26.566781 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:26.566744    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj" event={"ID":"7c066765-c46e-42e1-8d73-75bddd1dce76","Type":"ContainerDied","Data":"2c934749c8aee50c90441bed777f2af36ec07880a4ef85535ce8a04edcc3ab15"}
Nov 02 13:10:26.567247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:26.566800    5879 scope.go:122] "RemoveContainer" containerID="69e4ee0c3c44fb26bb164956b18a208280d4ee1ebbc768b5e7034667e3e21af3"
Nov 02 13:10:26.567247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:26.566752    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"
Nov 02 13:10:26.635386 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:26.635343    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"]
Nov 02 13:10:26.640197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:26.640165    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fbzdmqj"]
Nov 02 13:10:28.018883 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:28.018846    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c066765-c46e-42e1-8d73-75bddd1dce76" path="/var/lib/kubelet/pods/7c066765-c46e-42e1-8d73-75bddd1dce76/volumes"
Nov 02 13:10:28.102805 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:28.102774    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:29.828392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:29.828352    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:31.571150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:31.571118    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:31.714813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:31.714774    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:32.565326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:32.565285    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:32.700697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:32.700665    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:33.624425 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:33.624394    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:35.365034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:35.364984    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:37.259200 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:37.259170    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:38.576155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:38.576118    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:39.001716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:39.001678    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:40.782535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:40.782495    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:42.437341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:42.437305    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:44.172115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:44.172080    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:44.821880 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:10:44.821841    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:10:45.878155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:45.878118    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:46.098255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:46.098215    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"]
Nov 02 13:10:47.760523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:47.760488    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:49.446031 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:49.445980    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:51.098524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.098466    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv" podUID="6fc67eaf-0763-4be6-8db7-a52d2ae14a22" containerName="payload-processing" containerID="cri-o://dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5" gracePeriod=25
Nov 02 13:10:51.196215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.196184    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:51.405560 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.405532    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:10:51.523087 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.523058    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-plugins-config-volume\") pod \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") "
Nov 02 13:10:51.523289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.523139    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-provider-credentials\") pod \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") "
Nov 02 13:10:51.523423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.523401    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-kube-api-access-xdm6m\" (UniqueName: \"kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-kube-api-access-xdm6m\") pod \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") "
Nov 02 13:10:51.523481 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.523466    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-routing-overlay-volume\") pod \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\" (UID: \"6fc67eaf-0763-4be6-8db7-a52d2ae14a22\") "
Nov 02 13:10:51.523556 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.523530    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-plugins-config-volume" pod "6fc67eaf-0763-4be6-8db7-a52d2ae14a22" (UID: "6fc67eaf-0763-4be6-8db7-a52d2ae14a22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:51.523783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.523761    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-routing-overlay-volume" pod "6fc67eaf-0763-4be6-8db7-a52d2ae14a22" (UID: "6fc67eaf-0763-4be6-8db7-a52d2ae14a22"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:10:51.524071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.524054    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:51.524156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.524073    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:51.525612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.525585    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-provider-credentials" pod "6fc67eaf-0763-4be6-8db7-a52d2ae14a22" (UID: "6fc67eaf-0763-4be6-8db7-a52d2ae14a22"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:51.525735 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.525619    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-kube-api-access-xdm6m" pod "6fc67eaf-0763-4be6-8db7-a52d2ae14a22" (UID: "6fc67eaf-0763-4be6-8db7-a52d2ae14a22"). InnerVolumeSpecName "kube-api-access-xdm6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:10:51.625289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.625211    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdm6m\" (UniqueName: \"kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-kube-api-access-xdm6m\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:51.625289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.625239    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fc67eaf-0763-4be6-8db7-a52d2ae14a22-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:10:51.813654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.813622    5879 generic.go:356] "Generic (PLEG): container finished" podID="6fc67eaf-0763-4be6-8db7-a52d2ae14a22" containerID="dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5" exitCode=0
Nov 02 13:10:51.813859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.813683    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"
Nov 02 13:10:51.813859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.813709    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv" event={"ID":"6fc67eaf-0763-4be6-8db7-a52d2ae14a22","Type":"ContainerDied","Data":"dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5"}
Nov 02 13:10:51.813859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.813773    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv" event={"ID":"6fc67eaf-0763-4be6-8db7-a52d2ae14a22","Type":"ContainerDied","Data":"e7fffa2ccea7158b6d2997dedcca931096f389bded8cd490baf1086b1d3fb33d"}
Nov 02 13:10:51.813859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.813795    5879 scope.go:122] "RemoveContainer" containerID="dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5"
Nov 02 13:10:51.826108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.826087    5879 scope.go:122] "RemoveContainer" containerID="dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5"
Nov 02 13:10:51.826427 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:10:51.826403    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5\": container with ID starting with dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5 not found: ID does not exist" containerID="dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5"
Nov 02 13:10:51.826484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.826436    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5"} err="failed to get container status \"dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5\": rpc error: code = NotFound desc = could not find container \"dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5\": container with ID starting with dbe2c016349074b791b6a4a704cc5f58778efc071a46e94badf1884619f103c5 not found: ID does not exist"
Nov 02 13:10:51.923274 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.923185    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"]
Nov 02 13:10:51.943158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:51.943124    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bb658f5fc-8gkrv"]
Nov 02 13:10:52.019726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:52.019689    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc67eaf-0763-4be6-8db7-a52d2ae14a22" path="/var/lib/kubelet/pods/6fc67eaf-0763-4be6-8db7-a52d2ae14a22/volumes"
Nov 02 13:10:52.969548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:52.969516    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:54.671910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:54.671867    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:56.565605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:56.565569    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:10:58.670952 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:10:58.670910    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:00.577340 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:00.577301    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:01.473941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.473904    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"]
Nov 02 13:11:01.474701 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474686    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4c6522c4-39f2-4057-9e66-594e92c504a3" containerName="payload-processing"
Nov 02 13:11:01.474761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474703    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6522c4-39f2-4057-9e66-594e92c504a3" containerName="payload-processing"
Nov 02 13:11:01.474761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474739    5879 cpu_manager.go:414] "containerMap: removing container" podUID="b77c143f-8739-48c6-b37b-d6226bf36d0f" containerName="payload-processing"
Nov 02 13:11:01.474761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474745    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b77c143f-8739-48c6-b37b-d6226bf36d0f" containerName="payload-processing"
Nov 02 13:11:01.474761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474753    5879 cpu_manager.go:414] "containerMap: removing container" podUID="dcc6bf3f-d673-4095-bf6c-8a81b66fe351" containerName="payload-processing"
Nov 02 13:11:01.474761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474758    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc6bf3f-d673-4095-bf6c-8a81b66fe351" containerName="payload-processing"
Nov 02 13:11:01.474949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474772    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e0515ea2-a0ef-408b-a552-025591ee29de" containerName="revoke-keys"
Nov 02 13:11:01.474949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474778    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0515ea2-a0ef-408b-a552-025591ee29de" containerName="revoke-keys"
Nov 02 13:11:01.474949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474787    5879 cpu_manager.go:414] "containerMap: removing container" podUID="7c066765-c46e-42e1-8d73-75bddd1dce76" containerName="payload-processing"
Nov 02 13:11:01.474949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474791    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c066765-c46e-42e1-8d73-75bddd1dce76" containerName="payload-processing"
Nov 02 13:11:01.474949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474800    5879 cpu_manager.go:414] "containerMap: removing container" podUID="6fc67eaf-0763-4be6-8db7-a52d2ae14a22" containerName="payload-processing"
Nov 02 13:11:01.474949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474805    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc67eaf-0763-4be6-8db7-a52d2ae14a22" containerName="payload-processing"
Nov 02 13:11:01.474949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474821    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9123562e-d96a-4a86-9764-519429f8f5f6" containerName="payload-processing"
Nov 02 13:11:01.474949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474827    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9123562e-d96a-4a86-9764-519429f8f5f6" containerName="payload-processing"
Nov 02 13:11:01.474949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474944    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6522c4-39f2-4057-9e66-594e92c504a3" containerName="payload-processing"
Nov 02 13:11:01.475416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474954    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="b77c143f-8739-48c6-b37b-d6226bf36d0f" containerName="payload-processing"
Nov 02 13:11:01.475416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474963    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc6bf3f-d673-4095-bf6c-8a81b66fe351" containerName="payload-processing"
Nov 02 13:11:01.475416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474970    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0515ea2-a0ef-408b-a552-025591ee29de" containerName="revoke-keys"
Nov 02 13:11:01.475416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474976    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c066765-c46e-42e1-8d73-75bddd1dce76" containerName="payload-processing"
Nov 02 13:11:01.475416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.474984    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc67eaf-0763-4be6-8db7-a52d2ae14a22" containerName="payload-processing"
Nov 02 13:11:01.475416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.475007    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9123562e-d96a-4a86-9764-519429f8f5f6" containerName="payload-processing"
Nov 02 13:11:01.479332 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.479314    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:01.498313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.498282    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"]
Nov 02 13:11:01.505950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.505905    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:11:01.506132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.505959    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:11:01.522105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.522074    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ff2a4e6-mp6v9\" (UID: \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:01.522342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.522198    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ff2a4e6-mp6v9\" (UID: \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:01.522577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.522548    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc75q\" (UniqueName: \"kubernetes.io/projected/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-kube-api-access-bc75q\") pod \"maas-api-revoke-sub-0ff2a4e6-mp6v9\" (UID: \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:01.624125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.624089    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ff2a4e6-mp6v9\" (UID: \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:01.624646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.624168    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ff2a4e6-mp6v9\" (UID: \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:01.624646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.624338    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc75q\" (UniqueName: \"kubernetes.io/projected/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-kube-api-access-bc75q\") pod \"maas-api-revoke-sub-0ff2a4e6-mp6v9\" (UID: \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:01.624819 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.624796    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ff2a4e6-mp6v9\" (UID: \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:01.626841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.626821    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ff2a4e6-mp6v9\" (UID: \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:01.648947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.648920    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc75q\" (UniqueName: \"kubernetes.io/projected/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-kube-api-access-bc75q\") pod \"maas-api-revoke-sub-0ff2a4e6-mp6v9\" (UID: \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:01.795040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:01.794945    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:02.124815 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:11:02.124779    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda977c9b7_c0c7_4aeb_a5f2_53c190688e50.slice/crio-f758fbf279937726550db4274b172cdbe863e21b0db2d3544c5f1d71738bffd0 WatchSource:0}: Error finding container f758fbf279937726550db4274b172cdbe863e21b0db2d3544c5f1d71738bffd0: Status 404 returned error can't find the container with id f758fbf279937726550db4274b172cdbe863e21b0db2d3544c5f1d71738bffd0
Nov 02 13:11:02.125870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:02.125840    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"]
Nov 02 13:11:02.477271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:02.477215    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:02.932143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:02.932107    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9" event={"ID":"a977c9b7-c0c7-4aeb-a5f2-53c190688e50","Type":"ContainerStarted","Data":"d3a945eb645adb146fb96fd1fbcdc4497c4d4290a1c44d12c16020c39f6ad943"}
Nov 02 13:11:02.932577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:02.932177    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9" event={"ID":"a977c9b7-c0c7-4aeb-a5f2-53c190688e50","Type":"ContainerStarted","Data":"f758fbf279937726550db4274b172cdbe863e21b0db2d3544c5f1d71738bffd0"}
Nov 02 13:11:02.977861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:02.977805    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9" podStartSLOduration=1.977790785 podStartE2EDuration="1.977790785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:11:02.974150638 +0000 UTC m=+2609.080853488" watchObservedRunningTime="2026-11-02 13:11:02.977790785 +0000 UTC m=+2609.084493635"
Nov 02 13:11:03.944056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:03.944014    5879 generic.go:356] "Generic (PLEG): container finished" podID="a977c9b7-c0c7-4aeb-a5f2-53c190688e50" containerID="d3a945eb645adb146fb96fd1fbcdc4497c4d4290a1c44d12c16020c39f6ad943" exitCode=0
Nov 02 13:11:03.944465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:03.944107    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9" event={"ID":"a977c9b7-c0c7-4aeb-a5f2-53c190688e50","Type":"ContainerDied","Data":"d3a945eb645adb146fb96fd1fbcdc4497c4d4290a1c44d12c16020c39f6ad943"}
Nov 02 13:11:04.143642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:04.143579    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:05.125331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.125303    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:05.172864 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.172814    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-xks-ca\") pod \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\" (UID: \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\") "
Nov 02 13:11:05.173120 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.172923    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-kube-api-access-bc75q\" (UniqueName: \"kubernetes.io/projected/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-kube-api-access-bc75q\") pod \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\" (UID: \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\") "
Nov 02 13:11:05.173206 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.173176    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-openshift-ca\") pod \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\" (UID: \"a977c9b7-c0c7-4aeb-a5f2-53c190688e50\") "
Nov 02 13:11:05.174129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.174101    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-openshift-ca" pod "a977c9b7-c0c7-4aeb-a5f2-53c190688e50" (UID: "a977c9b7-c0c7-4aeb-a5f2-53c190688e50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:11:05.176165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.176137    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-xks-ca" pod "a977c9b7-c0c7-4aeb-a5f2-53c190688e50" (UID: "a977c9b7-c0c7-4aeb-a5f2-53c190688e50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:11:05.176557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.176536    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-kube-api-access-bc75q" pod "a977c9b7-c0c7-4aeb-a5f2-53c190688e50" (UID: "a977c9b7-c0c7-4aeb-a5f2-53c190688e50"). InnerVolumeSpecName "kube-api-access-bc75q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:11:05.274929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.274841    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:11:05.274929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.274874    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc75q\" (UniqueName: \"kubernetes.io/projected/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-kube-api-access-bc75q\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:11:05.274929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.274886    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a977c9b7-c0c7-4aeb-a5f2-53c190688e50-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:11:05.857472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.857432    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:05.963965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.963932    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"
Nov 02 13:11:05.964221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.963975    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9" event={"ID":"a977c9b7-c0c7-4aeb-a5f2-53c190688e50","Type":"ContainerDied","Data":"f758fbf279937726550db4274b172cdbe863e21b0db2d3544c5f1d71738bffd0"}
Nov 02 13:11:05.964221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:05.964048    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f758fbf279937726550db4274b172cdbe863e21b0db2d3544c5f1d71738bffd0"
Nov 02 13:11:07.673389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:07.673356    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:09.505894 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:09.505837    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:11.165487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:11.165412    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:12.692893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:12.692862    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:12.845169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:12.845132    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:13.178479 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:13.178441    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:14.581127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:14.581087    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:14.824403 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:11:14.824355    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:11:14.994385 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:14.994353    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:15.203689 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:15.203649    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:15.476523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:15.476485    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:16.972493 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:16.972457    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:18.893697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:18.893668    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:19.375129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:19.375092    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:19.527868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:19.527826    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:20.982314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:20.982278    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:22.425264 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:22.425230    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:22.913201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:22.913170    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:23.044847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:23.044077    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:23.186762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:23.186672    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:24.035923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:24.035888    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"]
Nov 02 13:11:24.047324 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:24.047297    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bd0e71-jtqc5"]
Nov 02 13:11:24.877536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:24.877496    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:25.768379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:25.768342    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:26.020085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:26.019954    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c" path="/var/lib/kubelet/pods/7fb9fa63-8ef6-4d2b-829b-be78b96a5a4c/volumes"
Nov 02 13:11:26.055527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:26.055488    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"]
Nov 02 13:11:26.060553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:26.060523    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2716652-2mlg2"]
Nov 02 13:11:26.761871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:26.761835    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:28.018300 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:28.018267    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594df756-ef67-40a2-b480-4168ec6c7ac0" path="/var/lib/kubelet/pods/594df756-ef67-40a2-b480-4168ec6c7ac0/volumes"
Nov 02 13:11:28.454331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:28.454296    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:30.074795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:30.074763    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:30.209553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:30.209519    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:30.374198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:30.374164    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:32.053895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:32.053865    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:33.891972 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:33.891936    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:34.042154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:34.042120    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"]
Nov 02 13:11:34.054836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:34.054803    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-222a3264-45fv9"]
Nov 02 13:11:35.605402 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:35.605367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:36.019677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:36.019639    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8916719-d328-48f3-b232-309806c64e7d" path="/var/lib/kubelet/pods/d8916719-d328-48f3-b232-309806c64e7d/volumes"
Nov 02 13:11:36.580673 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:36.580642    5879 scope.go:122] "RemoveContainer" containerID="32c87fbc1bc9eb50a6c4f1f4e14b47b5364fe4b97a69fa9be56adad4d47de63a"
Nov 02 13:11:37.441392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:37.441360    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:38.716394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:38.716356    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:39.168568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:39.168523    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:40.889699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:40.889660    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:42.698835 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:42.698799    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:44.458048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:44.457980    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:44.823614 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:11:44.823529    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:11:46.159812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:46.159779    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:47.039279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:47.039245    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"]
Nov 02 13:11:47.062934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:47.062898    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353ac4be-xv8pc"]
Nov 02 13:11:47.952561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:47.952516    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:48.020202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:48.020163    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1062baa0-a9ae-4b91-be5e-4c7d39e001bc" path="/var/lib/kubelet/pods/1062baa0-a9ae-4b91-be5e-4c7d39e001bc/volumes"
Nov 02 13:11:49.033248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:49.033210    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"]
Nov 02 13:11:49.046397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:49.046366    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af805d35-n66tk"]
Nov 02 13:11:49.695884 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:49.695848    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:50.018950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:50.018864    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76c829c-3682-4e2b-b53b-8a1f35260090" path="/var/lib/kubelet/pods/a76c829c-3682-4e2b-b53b-8a1f35260090/volumes"
Nov 02 13:11:51.598016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:51.597601    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:53.505019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:53.493101    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-27twq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:11:53.505019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:53.493185    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podUID="c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:11:53.548030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:53.527158    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-sg8t7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:11:53.548030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:53.527245    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:11:53.737017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:53.736861    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:55.570592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:55.570559    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:57.464499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:57.464465    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:11:59.375739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:11:59.375700    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:01.169239 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:01.169204    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:02.954693 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:02.954665    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:03.081523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:03.081492    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:04.785969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:04.785933    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:06.665479 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:06.665435    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:08.327580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:08.327545    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:09.880967 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:09.880935    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:11.653937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:11.653898    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:13.477843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:13.477809    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:14.824374 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:12:14.824329    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:12:15.339384 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:15.339355    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:16.978928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:16.978895    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:18.757023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:18.756969    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:20.361361 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:20.361326    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:22.049145 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:22.049109    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:23.818768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:23.818732    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:25.581184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:25.581146    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:27.392887 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:27.392846    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:28.961170 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:28.961136    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:30.568673 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:30.568635    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:32.143704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:32.143673    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:33.762134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:33.762099    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:35.405112 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:35.405072    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:36.615902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:36.615867    5879 scope.go:122] "RemoveContainer" containerID="8449eaef0c15b4068ec369c8ca816ab11cb2d15cc7907fa55f22c6e719db6bf3"
Nov 02 13:12:36.627129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:36.627109    5879 scope.go:122] "RemoveContainer" containerID="0e22335c6ad5d2714e07b4b8b4ce754593d556177ad30551a274fbb28275ba5c"
Nov 02 13:12:36.637701 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:36.637678    5879 scope.go:122] "RemoveContainer" containerID="8dd99fc52a055cf31c6c51b5d23ef3197920a2450d45795036f07a46b452c41f"
Nov 02 13:12:37.071217 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:37.071128    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:38.946592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:38.946547    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:39.085896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:39.085862    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:40.830843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:40.830813    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:42.659736 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:42.659683    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:43.941245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:43.941151    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"]
Nov 02 13:12:43.941981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:43.941965    5879 cpu_manager.go:414] "containerMap: removing container" podUID="a977c9b7-c0c7-4aeb-a5f2-53c190688e50" containerName="revoke-keys"
Nov 02 13:12:43.942089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:43.941982    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a977c9b7-c0c7-4aeb-a5f2-53c190688e50" containerName="revoke-keys"
Nov 02 13:12:43.942157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:43.942147    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="a977c9b7-c0c7-4aeb-a5f2-53c190688e50" containerName="revoke-keys"
Nov 02 13:12:43.947820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:43.947798    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:43.967810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:43.967772    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:12:43.968536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:43.968510    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsnmx\" (UniqueName: \"kubernetes.io/projected/3a91b416-95e1-45b3-ae59-e713edf8c562-kube-api-access-nsnmx\") pod \"maas-api-revoke-sub-c2a6a05a-87wvj\" (UID: \"3a91b416-95e1-45b3-ae59-e713edf8c562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:43.968651 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:43.968569    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a6a05a-87wvj\" (UID: \"3a91b416-95e1-45b3-ae59-e713edf8c562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:43.968651 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:43.968634    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a6a05a-87wvj\" (UID: \"3a91b416-95e1-45b3-ae59-e713edf8c562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:43.970749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:43.970725    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"]
Nov 02 13:12:43.978684 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:43.978652    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:12:44.070035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.069974    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsnmx\" (UniqueName: \"kubernetes.io/projected/3a91b416-95e1-45b3-ae59-e713edf8c562-kube-api-access-nsnmx\") pod \"maas-api-revoke-sub-c2a6a05a-87wvj\" (UID: \"3a91b416-95e1-45b3-ae59-e713edf8c562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:44.070273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.070069    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a6a05a-87wvj\" (UID: \"3a91b416-95e1-45b3-ae59-e713edf8c562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:44.070273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.070147    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a6a05a-87wvj\" (UID: \"3a91b416-95e1-45b3-ae59-e713edf8c562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:44.070761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.070738    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a6a05a-87wvj\" (UID: \"3a91b416-95e1-45b3-ae59-e713edf8c562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:44.072708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.072685    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a6a05a-87wvj\" (UID: \"3a91b416-95e1-45b3-ae59-e713edf8c562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:44.096165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.096131    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsnmx\" (UniqueName: \"kubernetes.io/projected/3a91b416-95e1-45b3-ae59-e713edf8c562-kube-api-access-nsnmx\") pod \"maas-api-revoke-sub-c2a6a05a-87wvj\" (UID: \"3a91b416-95e1-45b3-ae59-e713edf8c562\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:44.263555 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.263466    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:44.502406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.502366    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"]
Nov 02 13:12:44.508760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.508725    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:44.510192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.510162    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:44.539843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.539810    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"]
Nov 02 13:12:44.578075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.578021    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64975c55-7qh8l\" (UID: \"5fa1c331-dd6a-42d7-b210-5b50e9669626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:44.578550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.578456    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64975c55-7qh8l\" (UID: \"5fa1c331-dd6a-42d7-b210-5b50e9669626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:44.578550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.578500    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-992zh\" (UniqueName: \"kubernetes.io/projected/5fa1c331-dd6a-42d7-b210-5b50e9669626-kube-api-access-992zh\") pod \"maas-api-revoke-sub-64975c55-7qh8l\" (UID: \"5fa1c331-dd6a-42d7-b210-5b50e9669626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:44.586741 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.586714    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"]
Nov 02 13:12:44.680221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.680186    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64975c55-7qh8l\" (UID: \"5fa1c331-dd6a-42d7-b210-5b50e9669626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:44.680221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.680227    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-992zh\" (UniqueName: \"kubernetes.io/projected/5fa1c331-dd6a-42d7-b210-5b50e9669626-kube-api-access-992zh\") pod \"maas-api-revoke-sub-64975c55-7qh8l\" (UID: \"5fa1c331-dd6a-42d7-b210-5b50e9669626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:44.680483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.680256    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64975c55-7qh8l\" (UID: \"5fa1c331-dd6a-42d7-b210-5b50e9669626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:44.680966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.680938    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64975c55-7qh8l\" (UID: \"5fa1c331-dd6a-42d7-b210-5b50e9669626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:44.683092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.683060    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64975c55-7qh8l\" (UID: \"5fa1c331-dd6a-42d7-b210-5b50e9669626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:44.782425 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.782390    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-992zh\" (UniqueName: \"kubernetes.io/projected/5fa1c331-dd6a-42d7-b210-5b50e9669626-kube-api-access-992zh\") pod \"maas-api-revoke-sub-64975c55-7qh8l\" (UID: \"5fa1c331-dd6a-42d7-b210-5b50e9669626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:44.824251 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:12:44.824151    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:12:44.843688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.843653    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:44.998039 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:44.998003    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"]
Nov 02 13:12:45.004336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.004308    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:45.032861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.032830    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"]
Nov 02 13:12:45.033708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.033665    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj" event={"ID":"3a91b416-95e1-45b3-ae59-e713edf8c562","Type":"ContainerStarted","Data":"ffb32b55a3db43203461aa361a954f55aeb839aaccde633a8d4903784304c673"}
Nov 02 13:12:45.041033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.040947    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"]
Nov 02 13:12:45.048092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.048049    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c362144d-t9tpb"]
Nov 02 13:12:45.087397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.087351    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3358d95-603a-4760-a73b-a28daf972475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdec91b2-pchqc\" (UID: \"c3358d95-603a-4760-a73b-a28daf972475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:45.087581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.087527    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6sxm\" (UniqueName: \"kubernetes.io/projected/c3358d95-603a-4760-a73b-a28daf972475-kube-api-access-m6sxm\") pod \"maas-api-revoke-sub-cdec91b2-pchqc\" (UID: \"c3358d95-603a-4760-a73b-a28daf972475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:45.087705 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.087689    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3358d95-603a-4760-a73b-a28daf972475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdec91b2-pchqc\" (UID: \"c3358d95-603a-4760-a73b-a28daf972475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:45.158065 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:12:45.157981    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa1c331_dd6a_42d7_b210_5b50e9669626.slice/crio-ab5dc7c22ccc62899043e43c6f71f6342f6121eda1dac9ab7ac4b36d03669b30 WatchSource:0}: Error finding container ab5dc7c22ccc62899043e43c6f71f6342f6121eda1dac9ab7ac4b36d03669b30: Status 404 returned error can't find the container with id ab5dc7c22ccc62899043e43c6f71f6342f6121eda1dac9ab7ac4b36d03669b30
Nov 02 13:12:45.159971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.159945    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"]
Nov 02 13:12:45.189585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.189551    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3358d95-603a-4760-a73b-a28daf972475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdec91b2-pchqc\" (UID: \"c3358d95-603a-4760-a73b-a28daf972475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:45.189756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.189701    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3358d95-603a-4760-a73b-a28daf972475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdec91b2-pchqc\" (UID: \"c3358d95-603a-4760-a73b-a28daf972475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:45.189820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.189803    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6sxm\" (UniqueName: \"kubernetes.io/projected/c3358d95-603a-4760-a73b-a28daf972475-kube-api-access-m6sxm\") pod \"maas-api-revoke-sub-cdec91b2-pchqc\" (UID: \"c3358d95-603a-4760-a73b-a28daf972475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:45.190487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.190405    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3358d95-603a-4760-a73b-a28daf972475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdec91b2-pchqc\" (UID: \"c3358d95-603a-4760-a73b-a28daf972475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:45.192282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.192257    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3358d95-603a-4760-a73b-a28daf972475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdec91b2-pchqc\" (UID: \"c3358d95-603a-4760-a73b-a28daf972475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:45.215794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.215764    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6sxm\" (UniqueName: \"kubernetes.io/projected/c3358d95-603a-4760-a73b-a28daf972475-kube-api-access-m6sxm\") pod \"maas-api-revoke-sub-cdec91b2-pchqc\" (UID: \"c3358d95-603a-4760-a73b-a28daf972475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:45.320585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.320548    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:45.637286 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:12:45.637183    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3358d95_603a_4760_a73b_a28daf972475.slice/crio-9a55d2d3e42c0ef2d41b8db6486cfec3bd8ad3f3f8db2bf0ca5434c81b9d226e WatchSource:0}: Error finding container 9a55d2d3e42c0ef2d41b8db6486cfec3bd8ad3f3f8db2bf0ca5434c81b9d226e: Status 404 returned error can't find the container with id 9a55d2d3e42c0ef2d41b8db6486cfec3bd8ad3f3f8db2bf0ca5434c81b9d226e
Nov 02 13:12:45.639667 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:45.639636    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"]
Nov 02 13:12:46.019174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:46.019139    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95442b0e-e7b1-403f-9ad3-a6121c7b6b95" path="/var/lib/kubelet/pods/95442b0e-e7b1-403f-9ad3-a6121c7b6b95/volumes"
Nov 02 13:12:46.043772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:46.043731    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l" event={"ID":"5fa1c331-dd6a-42d7-b210-5b50e9669626","Type":"ContainerStarted","Data":"b8e98c8044bdebcbbcc04d168d15c50e3d7e227071872a0e9237056a00b34800"}
Nov 02 13:12:46.043985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:46.043804    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l" event={"ID":"5fa1c331-dd6a-42d7-b210-5b50e9669626","Type":"ContainerStarted","Data":"ab5dc7c22ccc62899043e43c6f71f6342f6121eda1dac9ab7ac4b36d03669b30"}
Nov 02 13:12:46.045679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:46.045647    5879 generic.go:356] "Generic (PLEG): container finished" podID="3a91b416-95e1-45b3-ae59-e713edf8c562" containerID="d18c81f7c165136af4755cf22fb8d5f2edcec5288724dfc0c045c12df55a2e5f" exitCode=0
Nov 02 13:12:46.045839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:46.045735    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj" event={"ID":"3a91b416-95e1-45b3-ae59-e713edf8c562","Type":"ContainerDied","Data":"d18c81f7c165136af4755cf22fb8d5f2edcec5288724dfc0c045c12df55a2e5f"}
Nov 02 13:12:46.047213 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:46.047190    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc" event={"ID":"c3358d95-603a-4760-a73b-a28daf972475","Type":"ContainerStarted","Data":"9a55d2d3e42c0ef2d41b8db6486cfec3bd8ad3f3f8db2bf0ca5434c81b9d226e"}
Nov 02 13:12:46.090101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:46.089522    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l" podStartSLOduration=2.089505734 podStartE2EDuration="2.089505734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:46.086977765 +0000 UTC m=+2712.193680617" watchObservedRunningTime="2026-11-02 13:12:46.089505734 +0000 UTC m=+2712.196208585"
Nov 02 13:12:46.278688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:46.278592    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:47.057165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.057074    5879 generic.go:356] "Generic (PLEG): container finished" podID="5fa1c331-dd6a-42d7-b210-5b50e9669626" containerID="b8e98c8044bdebcbbcc04d168d15c50e3d7e227071872a0e9237056a00b34800" exitCode=0
Nov 02 13:12:47.057577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.057165    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l" event={"ID":"5fa1c331-dd6a-42d7-b210-5b50e9669626","Type":"ContainerDied","Data":"b8e98c8044bdebcbbcc04d168d15c50e3d7e227071872a0e9237056a00b34800"}
Nov 02 13:12:47.058890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.058867    5879 generic.go:356] "Generic (PLEG): container finished" podID="c3358d95-603a-4760-a73b-a28daf972475" containerID="e26f28108c91c5d95f770669b044ad255bc0d3afd6265f9f1d8a62dc02059d7f" exitCode=0
Nov 02 13:12:47.058976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.058946    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc" event={"ID":"c3358d95-603a-4760-a73b-a28daf972475","Type":"ContainerDied","Data":"e26f28108c91c5d95f770669b044ad255bc0d3afd6265f9f1d8a62dc02059d7f"}
Nov 02 13:12:47.248588 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.248559    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:47.319111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.319016    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-openshift-ca\") pod \"3a91b416-95e1-45b3-ae59-e713edf8c562\" (UID: \"3a91b416-95e1-45b3-ae59-e713edf8c562\") "
Nov 02 13:12:47.319430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.319194    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-xks-ca\") pod \"3a91b416-95e1-45b3-ae59-e713edf8c562\" (UID: \"3a91b416-95e1-45b3-ae59-e713edf8c562\") "
Nov 02 13:12:47.319430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.319370    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a91b416-95e1-45b3-ae59-e713edf8c562-kube-api-access-nsnmx\" (UniqueName: \"kubernetes.io/projected/3a91b416-95e1-45b3-ae59-e713edf8c562-kube-api-access-nsnmx\") pod \"3a91b416-95e1-45b3-ae59-e713edf8c562\" (UID: \"3a91b416-95e1-45b3-ae59-e713edf8c562\") "
Nov 02 13:12:47.319653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.319493    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-openshift-ca" pod "3a91b416-95e1-45b3-ae59-e713edf8c562" (UID: "3a91b416-95e1-45b3-ae59-e713edf8c562"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:47.320032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.320010    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:47.321751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.321727    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-xks-ca" pod "3a91b416-95e1-45b3-ae59-e713edf8c562" (UID: "3a91b416-95e1-45b3-ae59-e713edf8c562"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:12:47.321948 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.321930    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a91b416-95e1-45b3-ae59-e713edf8c562-kube-api-access-nsnmx" pod "3a91b416-95e1-45b3-ae59-e713edf8c562" (UID: "3a91b416-95e1-45b3-ae59-e713edf8c562"). InnerVolumeSpecName "kube-api-access-nsnmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:47.421619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.421569    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsnmx\" (UniqueName: \"kubernetes.io/projected/3a91b416-95e1-45b3-ae59-e713edf8c562-kube-api-access-nsnmx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:47.421619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:47.421610    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a91b416-95e1-45b3-ae59-e713edf8c562-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:48.067844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.067809    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:48.068881 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.068848    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj" event={"ID":"3a91b416-95e1-45b3-ae59-e713edf8c562","Type":"ContainerDied","Data":"ffb32b55a3db43203461aa361a954f55aeb839aaccde633a8d4903784304c673"}
Nov 02 13:12:48.068881 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.068875    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"
Nov 02 13:12:48.069138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.068903    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffb32b55a3db43203461aa361a954f55aeb839aaccde633a8d4903784304c673"
Nov 02 13:12:48.287310 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.287228    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:48.340256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.336036    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:48.340448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.340420    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa1c331-dd6a-42d7-b210-5b50e9669626-kube-api-access-992zh\" (UniqueName: \"kubernetes.io/projected/5fa1c331-dd6a-42d7-b210-5b50e9669626-kube-api-access-992zh\") pod \"5fa1c331-dd6a-42d7-b210-5b50e9669626\" (UID: \"5fa1c331-dd6a-42d7-b210-5b50e9669626\") "
Nov 02 13:12:48.340556 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.340531    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-openshift-ca\") pod \"5fa1c331-dd6a-42d7-b210-5b50e9669626\" (UID: \"5fa1c331-dd6a-42d7-b210-5b50e9669626\") "
Nov 02 13:12:48.340628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.340612    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-xks-ca\") pod \"5fa1c331-dd6a-42d7-b210-5b50e9669626\" (UID: \"5fa1c331-dd6a-42d7-b210-5b50e9669626\") "
Nov 02 13:12:48.342270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.342242    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-openshift-ca" pod "5fa1c331-dd6a-42d7-b210-5b50e9669626" (UID: "5fa1c331-dd6a-42d7-b210-5b50e9669626"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:48.348656 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.348620    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa1c331-dd6a-42d7-b210-5b50e9669626-kube-api-access-992zh" pod "5fa1c331-dd6a-42d7-b210-5b50e9669626" (UID: "5fa1c331-dd6a-42d7-b210-5b50e9669626"). InnerVolumeSpecName "kube-api-access-992zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:48.355681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.355618    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-xks-ca" pod "5fa1c331-dd6a-42d7-b210-5b50e9669626" (UID: "5fa1c331-dd6a-42d7-b210-5b50e9669626"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:12:48.442068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.442024    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3358d95-603a-4760-a73b-a28daf972475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3358d95-603a-4760-a73b-a28daf972475-maas-api-xks-ca\") pod \"c3358d95-603a-4760-a73b-a28daf972475\" (UID: \"c3358d95-603a-4760-a73b-a28daf972475\") "
Nov 02 13:12:48.442232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.442119    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3358d95-603a-4760-a73b-a28daf972475-kube-api-access-m6sxm\" (UniqueName: \"kubernetes.io/projected/c3358d95-603a-4760-a73b-a28daf972475-kube-api-access-m6sxm\") pod \"c3358d95-603a-4760-a73b-a28daf972475\" (UID: \"c3358d95-603a-4760-a73b-a28daf972475\") "
Nov 02 13:12:48.442295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.442276    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3358d95-603a-4760-a73b-a28daf972475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3358d95-603a-4760-a73b-a28daf972475-maas-api-openshift-ca\") pod \"c3358d95-603a-4760-a73b-a28daf972475\" (UID: \"c3358d95-603a-4760-a73b-a28daf972475\") "
Nov 02 13:12:48.442772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.442691    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3358d95-603a-4760-a73b-a28daf972475-maas-api-openshift-ca" pod "c3358d95-603a-4760-a73b-a28daf972475" (UID: "c3358d95-603a-4760-a73b-a28daf972475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:48.443097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.443052    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3358d95-603a-4760-a73b-a28daf972475-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:48.443097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.443077    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-992zh\" (UniqueName: \"kubernetes.io/projected/5fa1c331-dd6a-42d7-b210-5b50e9669626-kube-api-access-992zh\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:48.443097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.443091    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:48.443314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.443106    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fa1c331-dd6a-42d7-b210-5b50e9669626-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:48.444508 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.444479    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3358d95-603a-4760-a73b-a28daf972475-maas-api-xks-ca" pod "c3358d95-603a-4760-a73b-a28daf972475" (UID: "c3358d95-603a-4760-a73b-a28daf972475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:12:48.444645 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.444524    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3358d95-603a-4760-a73b-a28daf972475-kube-api-access-m6sxm" pod "c3358d95-603a-4760-a73b-a28daf972475" (UID: "c3358d95-603a-4760-a73b-a28daf972475"). InnerVolumeSpecName "kube-api-access-m6sxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:48.544980 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.544898    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6sxm\" (UniqueName: \"kubernetes.io/projected/c3358d95-603a-4760-a73b-a28daf972475-kube-api-access-m6sxm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:48.544980 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:48.544934    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3358d95-603a-4760-a73b-a28daf972475-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:49.078698 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:49.078665    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"
Nov 02 13:12:49.079191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:49.078696    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l" event={"ID":"5fa1c331-dd6a-42d7-b210-5b50e9669626","Type":"ContainerDied","Data":"ab5dc7c22ccc62899043e43c6f71f6342f6121eda1dac9ab7ac4b36d03669b30"}
Nov 02 13:12:49.079191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:49.078746    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab5dc7c22ccc62899043e43c6f71f6342f6121eda1dac9ab7ac4b36d03669b30"
Nov 02 13:12:49.080328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:49.080309    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"
Nov 02 13:12:49.080434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:49.080353    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc" event={"ID":"c3358d95-603a-4760-a73b-a28daf972475","Type":"ContainerDied","Data":"9a55d2d3e42c0ef2d41b8db6486cfec3bd8ad3f3f8db2bf0ca5434c81b9d226e"}
Nov 02 13:12:49.080434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:49.080403    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a55d2d3e42c0ef2d41b8db6486cfec3bd8ad3f3f8db2bf0ca5434c81b9d226e"
Nov 02 13:12:49.939125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:49.939092    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:51.400532 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:51.400496    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:51.579341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:51.579306    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:51.804644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:51.804558    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:52.268512 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:52.268445    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:52.404030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:52.398878    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:53.230019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:53.229287    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:54.864007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:54.863966    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:55.327983 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.327457    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:55.445073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.445041    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:55.507502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.507461    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"]
Nov 02 13:12:55.508323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.508301    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5fa1c331-dd6a-42d7-b210-5b50e9669626" containerName="revoke-keys"
Nov 02 13:12:55.508398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.508325    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa1c331-dd6a-42d7-b210-5b50e9669626" containerName="revoke-keys"
Nov 02 13:12:55.508438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.508412    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3a91b416-95e1-45b3-ae59-e713edf8c562" containerName="revoke-keys"
Nov 02 13:12:55.508438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.508418    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a91b416-95e1-45b3-ae59-e713edf8c562" containerName="revoke-keys"
Nov 02 13:12:55.508438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.508436    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c3358d95-603a-4760-a73b-a28daf972475" containerName="revoke-keys"
Nov 02 13:12:55.508540 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.508442    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3358d95-603a-4760-a73b-a28daf972475" containerName="revoke-keys"
Nov 02 13:12:55.508579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.508557    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa1c331-dd6a-42d7-b210-5b50e9669626" containerName="revoke-keys"
Nov 02 13:12:55.508579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.508574    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a91b416-95e1-45b3-ae59-e713edf8c562" containerName="revoke-keys"
Nov 02 13:12:55.508683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.508585    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3358d95-603a-4760-a73b-a28daf972475" containerName="revoke-keys"
Nov 02 13:12:55.513193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.513164    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:12:55.531478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.531444    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"]
Nov 02 13:12:55.539490 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.539460    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:12:55.539755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.539599    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:12:55.676033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.675923    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjjx6\" (UniqueName: \"kubernetes.io/projected/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-kube-api-access-rjjx6\") pod \"maas-api-revoke-sub-08754800-wl8gj\" (UID: \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:12:55.676276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.676087    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08754800-wl8gj\" (UID: \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:12:55.676352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.676271    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08754800-wl8gj\" (UID: \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:12:55.778284 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.778246    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08754800-wl8gj\" (UID: \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:12:55.778558 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.778535    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjjx6\" (UniqueName: \"kubernetes.io/projected/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-kube-api-access-rjjx6\") pod \"maas-api-revoke-sub-08754800-wl8gj\" (UID: \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:12:55.778655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.778604    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08754800-wl8gj\" (UID: \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:12:55.779093 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.779068    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08754800-wl8gj\" (UID: \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:12:55.781398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.781376    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08754800-wl8gj\" (UID: \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:12:55.806144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.806111    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjjx6\" (UniqueName: \"kubernetes.io/projected/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-kube-api-access-rjjx6\") pod \"maas-api-revoke-sub-08754800-wl8gj\" (UID: \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:12:55.832220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.832185    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:12:55.908438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.908406    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"]
Nov 02 13:12:55.919598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.919573    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:12:55.936854 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:55.936688    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"]
Nov 02 13:12:56.084050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.083986    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82853549-d7zk2\" (UID: \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:12:56.084289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.084262    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jr8r\" (UniqueName: \"kubernetes.io/projected/577acb2a-8bfc-4443-8e71-600fc40e5d1b-kube-api-access-6jr8r\") pod \"maas-api-revoke-sub-82853549-d7zk2\" (UID: \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:12:56.084359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.084325    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82853549-d7zk2\" (UID: \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:12:56.149396 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:12:56.149361    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb1da1a_2eb8_4d89_a8f6_11d75673270b.slice/crio-60b2b0bb3f1032583754897719184424e0c940e91d4207aeea8dd69492a23a6f WatchSource:0}: Error finding container 60b2b0bb3f1032583754897719184424e0c940e91d4207aeea8dd69492a23a6f: Status 404 returned error can't find the container with id 60b2b0bb3f1032583754897719184424e0c940e91d4207aeea8dd69492a23a6f
Nov 02 13:12:56.150478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.150450    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"]
Nov 02 13:12:56.186241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.186199    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jr8r\" (UniqueName: \"kubernetes.io/projected/577acb2a-8bfc-4443-8e71-600fc40e5d1b-kube-api-access-6jr8r\") pod \"maas-api-revoke-sub-82853549-d7zk2\" (UID: \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:12:56.186443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.186275    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82853549-d7zk2\" (UID: \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:12:56.186517 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.186474    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82853549-d7zk2\" (UID: \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:12:56.187227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.187166    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82853549-d7zk2\" (UID: \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:12:56.189083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.189051    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82853549-d7zk2\" (UID: \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:12:56.211883 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.211852    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jr8r\" (UniqueName: \"kubernetes.io/projected/577acb2a-8bfc-4443-8e71-600fc40e5d1b-kube-api-access-6jr8r\") pod \"maas-api-revoke-sub-82853549-d7zk2\" (UID: \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:12:56.253933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.253904    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:12:56.664341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.664306    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"]
Nov 02 13:12:56.740298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.740254    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:56.917360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:56.917324    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:57.064307 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:57.064273    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:57.154939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:57.154848    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj" event={"ID":"6eb1da1a-2eb8-4d89-a8f6-11d75673270b","Type":"ContainerStarted","Data":"e561ed463e1d27651d83564212455963fb1ddd9ab5e14efa832a93259d1b62ff"}
Nov 02 13:12:57.154939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:57.154904    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj" event={"ID":"6eb1da1a-2eb8-4d89-a8f6-11d75673270b","Type":"ContainerStarted","Data":"60b2b0bb3f1032583754897719184424e0c940e91d4207aeea8dd69492a23a6f"}
Nov 02 13:12:57.156796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:57.156767    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2" event={"ID":"577acb2a-8bfc-4443-8e71-600fc40e5d1b","Type":"ContainerStarted","Data":"5f61951e5143d2de896f5fce23ad77228297abc133c0413f17a5f6b0da578a02"}
Nov 02 13:12:57.156941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:57.156819    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2" event={"ID":"577acb2a-8bfc-4443-8e71-600fc40e5d1b","Type":"ContainerStarted","Data":"9ab92fe565e8a59a41692f2d545803662fd8fc1e628ef2f4f6d085d3b306c695"}
Nov 02 13:12:57.203868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:57.203833    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:57.214854 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:57.214791    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj" podStartSLOduration=2.214771007 podStartE2EDuration="2.214771007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:57.212251278 +0000 UTC m=+2723.318954129" watchObservedRunningTime="2026-11-02 13:12:57.214771007 +0000 UTC m=+2723.321473859"
Nov 02 13:12:57.261271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:57.261207    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2" podStartSLOduration=2.26117781 podStartE2EDuration="2.26117781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:12:57.256691451 +0000 UTC m=+2723.363394304" watchObservedRunningTime="2026-11-02 13:12:57.26117781 +0000 UTC m=+2723.367880660"
Nov 02 13:12:57.335248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:57.335216    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:58.038198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:58.038113    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"]
Nov 02 13:12:58.051929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:58.051894    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4bec90-sbgbw"]
Nov 02 13:12:58.166811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:58.166768    5879 generic.go:356] "Generic (PLEG): container finished" podID="577acb2a-8bfc-4443-8e71-600fc40e5d1b" containerID="5f61951e5143d2de896f5fce23ad77228297abc133c0413f17a5f6b0da578a02" exitCode=0
Nov 02 13:12:58.167032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:58.166851    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2" event={"ID":"577acb2a-8bfc-4443-8e71-600fc40e5d1b","Type":"ContainerDied","Data":"5f61951e5143d2de896f5fce23ad77228297abc133c0413f17a5f6b0da578a02"}
Nov 02 13:12:58.168711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:58.168687    5879 generic.go:356] "Generic (PLEG): container finished" podID="6eb1da1a-2eb8-4d89-a8f6-11d75673270b" containerID="e561ed463e1d27651d83564212455963fb1ddd9ab5e14efa832a93259d1b62ff" exitCode=0
Nov 02 13:12:58.168837 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:58.168763    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj" event={"ID":"6eb1da1a-2eb8-4d89-a8f6-11d75673270b","Type":"ContainerDied","Data":"e561ed463e1d27651d83564212455963fb1ddd9ab5e14efa832a93259d1b62ff"}
Nov 02 13:12:58.989602 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:58.989567    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:59.205039 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.203098    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:59.503299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.503268    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:12:59.507172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.507142    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:12:59.606130 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.600521    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:59.699318 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.699284    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-kube-api-access-rjjx6\" (UniqueName: \"kubernetes.io/projected/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-kube-api-access-rjjx6\") pod \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\" (UID: \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\") "
Nov 02 13:12:59.701146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.701119    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-xks-ca\") pod \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\" (UID: \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\") "
Nov 02 13:12:59.701371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.701353    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-xks-ca\") pod \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\" (UID: \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\") "
Nov 02 13:12:59.701687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.701668    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-openshift-ca\") pod \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\" (UID: \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\") "
Nov 02 13:12:59.701851 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.701834    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577acb2a-8bfc-4443-8e71-600fc40e5d1b-kube-api-access-6jr8r\" (UniqueName: \"kubernetes.io/projected/577acb2a-8bfc-4443-8e71-600fc40e5d1b-kube-api-access-6jr8r\") pod \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\" (UID: \"577acb2a-8bfc-4443-8e71-600fc40e5d1b\") "
Nov 02 13:12:59.702218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.702198    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-openshift-ca\") pod \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\" (UID: \"6eb1da1a-2eb8-4d89-a8f6-11d75673270b\") "
Nov 02 13:12:59.704578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.703371    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-openshift-ca" pod "577acb2a-8bfc-4443-8e71-600fc40e5d1b" (UID: "577acb2a-8bfc-4443-8e71-600fc40e5d1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:59.704891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.704870    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-openshift-ca" pod "6eb1da1a-2eb8-4d89-a8f6-11d75673270b" (UID: "6eb1da1a-2eb8-4d89-a8f6-11d75673270b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:12:59.707092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.707065    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-xks-ca" pod "577acb2a-8bfc-4443-8e71-600fc40e5d1b" (UID: "577acb2a-8bfc-4443-8e71-600fc40e5d1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:12:59.732552 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.732509    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-xks-ca" pod "6eb1da1a-2eb8-4d89-a8f6-11d75673270b" (UID: "6eb1da1a-2eb8-4d89-a8f6-11d75673270b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:12:59.732778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.732747    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-kube-api-access-rjjx6" pod "6eb1da1a-2eb8-4d89-a8f6-11d75673270b" (UID: "6eb1da1a-2eb8-4d89-a8f6-11d75673270b"). InnerVolumeSpecName "kube-api-access-rjjx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:59.735323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.735153    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577acb2a-8bfc-4443-8e71-600fc40e5d1b-kube-api-access-6jr8r" pod "577acb2a-8bfc-4443-8e71-600fc40e5d1b" (UID: "577acb2a-8bfc-4443-8e71-600fc40e5d1b"). InnerVolumeSpecName "kube-api-access-6jr8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:12:59.804176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.804141    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:59.804176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.804174    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jr8r\" (UniqueName: \"kubernetes.io/projected/577acb2a-8bfc-4443-8e71-600fc40e5d1b-kube-api-access-6jr8r\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:59.804176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.804190    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:59.804459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.804202    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjjx6\" (UniqueName: \"kubernetes.io/projected/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-kube-api-access-rjjx6\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:59.804459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.804215    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577acb2a-8bfc-4443-8e71-600fc40e5d1b-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:59.804459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.804227    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6eb1da1a-2eb8-4d89-a8f6-11d75673270b-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:12:59.844325 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.844292    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:12:59.993368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:12:59.993320    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:00.020817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.020781    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb" path="/var/lib/kubelet/pods/23a74ef2-5ad2-4e6d-964c-bf6094fc5ffb/volumes"
Nov 02 13:13:00.082894 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.082854    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"]
Nov 02 13:13:00.084141 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.084113    5879 cpu_manager.go:414] "containerMap: removing container" podUID="577acb2a-8bfc-4443-8e71-600fc40e5d1b" containerName="revoke-keys"
Nov 02 13:13:00.084141 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.084141    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577acb2a-8bfc-4443-8e71-600fc40e5d1b" containerName="revoke-keys"
Nov 02 13:13:00.084353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.084223    5879 cpu_manager.go:414] "containerMap: removing container" podUID="6eb1da1a-2eb8-4d89-a8f6-11d75673270b" containerName="revoke-keys"
Nov 02 13:13:00.084353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.084231    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb1da1a-2eb8-4d89-a8f6-11d75673270b" containerName="revoke-keys"
Nov 02 13:13:00.084463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.084433    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb1da1a-2eb8-4d89-a8f6-11d75673270b" containerName="revoke-keys"
Nov 02 13:13:00.084463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.084453    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="577acb2a-8bfc-4443-8e71-600fc40e5d1b" containerName="revoke-keys"
Nov 02 13:13:00.089742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.089714    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"]
Nov 02 13:13:00.089911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.089842    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"
Nov 02 13:13:00.094881 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.094854    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"
Nov 02 13:13:00.109298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.109241    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"]
Nov 02 13:13:00.111399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.111369    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9139ec\""
Nov 02 13:13:00.111568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.111483    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9139ec-dockercfg-kcsfp\""
Nov 02 13:13:00.116041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.116008    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"]
Nov 02 13:13:00.190793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.190759    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"
Nov 02 13:13:00.191060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.190790    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2" event={"ID":"577acb2a-8bfc-4443-8e71-600fc40e5d1b","Type":"ContainerDied","Data":"9ab92fe565e8a59a41692f2d545803662fd8fc1e628ef2f4f6d085d3b306c695"}
Nov 02 13:13:00.191060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.190849    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ab92fe565e8a59a41692f2d545803662fd8fc1e628ef2f4f6d085d3b306c695"
Nov 02 13:13:00.192606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.192571    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"
Nov 02 13:13:00.192748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.192617    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj" event={"ID":"6eb1da1a-2eb8-4d89-a8f6-11d75673270b","Type":"ContainerDied","Data":"60b2b0bb3f1032583754897719184424e0c940e91d4207aeea8dd69492a23a6f"}
Nov 02 13:13:00.192748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.192660    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60b2b0bb3f1032583754897719184424e0c940e91d4207aeea8dd69492a23a6f"
Nov 02 13:13:00.216055 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.215944    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6d6z\" (UniqueName: \"kubernetes.io/projected/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-kube-api-access-l6d6z\") pod \"payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6\" (UID: \"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"
Nov 02 13:13:00.216738 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.216705    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vshdk\" (UniqueName: \"kubernetes.io/projected/abe4af2d-1652-47e4-8883-0ca27eb781db-kube-api-access-vshdk\") pod \"payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz\" (UID: \"abe4af2d-1652-47e4-8883-0ca27eb781db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"
Nov 02 13:13:00.216946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.216921    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6\" (UID: \"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"
Nov 02 13:13:00.217122 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.217099    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe4af2d-1652-47e4-8883-0ca27eb781db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz\" (UID: \"abe4af2d-1652-47e4-8883-0ca27eb781db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"
Nov 02 13:13:00.239188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.239147    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:00.334030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.330513    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6\" (UID: \"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"
Nov 02 13:13:00.334030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.331530    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6\" (UID: \"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"
Nov 02 13:13:00.334030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.331802    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe4af2d-1652-47e4-8883-0ca27eb781db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz\" (UID: \"abe4af2d-1652-47e4-8883-0ca27eb781db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"
Nov 02 13:13:00.334030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.331876    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6d6z\" (UniqueName: \"kubernetes.io/projected/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-kube-api-access-l6d6z\") pod \"payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6\" (UID: \"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"
Nov 02 13:13:00.352019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.344289    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshdk\" (UniqueName: \"kubernetes.io/projected/abe4af2d-1652-47e4-8883-0ca27eb781db-kube-api-access-vshdk\") pod \"payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz\" (UID: \"abe4af2d-1652-47e4-8883-0ca27eb781db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"
Nov 02 13:13:00.352019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.345489    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe4af2d-1652-47e4-8883-0ca27eb781db-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz\" (UID: \"abe4af2d-1652-47e4-8883-0ca27eb781db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"
Nov 02 13:13:00.383379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.383340    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshdk\" (UniqueName: \"kubernetes.io/projected/abe4af2d-1652-47e4-8883-0ca27eb781db-kube-api-access-vshdk\") pod \"payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz\" (UID: \"abe4af2d-1652-47e4-8883-0ca27eb781db\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"
Nov 02 13:13:00.383641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.383555    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6d6z\" (UniqueName: \"kubernetes.io/projected/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-kube-api-access-l6d6z\") pod \"payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6\" (UID: \"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"
Nov 02 13:13:00.412301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.412262    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"
Nov 02 13:13:00.437733 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.437703    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"
Nov 02 13:13:00.732620 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:13:00.732586    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc45a3d_f7bf_43a3_a0a1_daa8bd1bb89a.slice/crio-cf54e928710891ecf80a217eff72f984854f9c1e84fc2eff48a35525d1186260 WatchSource:0}: Error finding container cf54e928710891ecf80a217eff72f984854f9c1e84fc2eff48a35525d1186260: Status 404 returned error can't find the container with id cf54e928710891ecf80a217eff72f984854f9c1e84fc2eff48a35525d1186260
Nov 02 13:13:00.734739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.734716    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"]
Nov 02 13:13:00.776341 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:13:00.774759    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe4af2d_1652_47e4_8883_0ca27eb781db.slice/crio-8e4a41f777ec6dda194a9fd86f6294c16ec7cd724c37ec4caed2640aada539dc WatchSource:0}: Error finding container 8e4a41f777ec6dda194a9fd86f6294c16ec7cd724c37ec4caed2640aada539dc: Status 404 returned error can't find the container with id 8e4a41f777ec6dda194a9fd86f6294c16ec7cd724c37ec4caed2640aada539dc
Nov 02 13:13:00.776341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:00.775058    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"]
Nov 02 13:13:01.152738 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:01.152695    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:01.217119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:01.217082    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz" event={"ID":"abe4af2d-1652-47e4-8883-0ca27eb781db","Type":"ContainerStarted","Data":"fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd"}
Nov 02 13:13:01.217639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:01.217609    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"
Nov 02 13:13:01.217758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:01.217644    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz" event={"ID":"abe4af2d-1652-47e4-8883-0ca27eb781db","Type":"ContainerStarted","Data":"8e4a41f777ec6dda194a9fd86f6294c16ec7cd724c37ec4caed2640aada539dc"}
Nov 02 13:13:01.219310 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:01.219286    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6" event={"ID":"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a","Type":"ContainerStarted","Data":"0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49"}
Nov 02 13:13:01.219448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:01.219335    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6" event={"ID":"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a","Type":"ContainerStarted","Data":"cf54e928710891ecf80a217eff72f984854f9c1e84fc2eff48a35525d1186260"}
Nov 02 13:13:01.219448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:01.219364    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"
Nov 02 13:13:01.268801 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:01.268736    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz" podStartSLOduration=1.268715235 podStartE2EDuration="1.268715235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:01.265300699 +0000 UTC m=+2727.372003551" watchObservedRunningTime="2026-11-02 13:13:01.268715235 +0000 UTC m=+2727.375418086"
Nov 02 13:13:01.314538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:01.314481    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6" podStartSLOduration=1.314465933 podStartE2EDuration="1.314465933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:01.308714924 +0000 UTC m=+2727.415417777" watchObservedRunningTime="2026-11-02 13:13:01.314465933 +0000 UTC m=+2727.421168784"
Nov 02 13:13:02.900742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:02.900698    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:03.889105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:03.889070    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:04.040598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:04.040564    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:04.729646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:04.729610    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:05.564420 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:05.564384    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:05.686576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:05.686542    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:06.441692 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:06.441651    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:07.279330 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:07.279287    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:07.424080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:07.422605    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:08.264533 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.264496    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:08.464293 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.464257    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:08.614857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.614799    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"]
Nov 02 13:13:08.633687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.633619    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:08.643018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.639490    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"]
Nov 02 13:13:08.643018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.639622    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:08.813704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.813662    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:08.814130 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.814102    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:08.814244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.814178    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:08.814424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.814392    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzjvn\" (UniqueName: \"kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-kube-api-access-zzjvn\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:08.927578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.926045    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:08.927868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.927842    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:08.928128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.928107    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:08.928390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.928370    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzjvn\" (UniqueName: \"kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-kube-api-access-zzjvn\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:08.928526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.928497    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:08.928804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.927401    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:08.928938 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:08.928751    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-00d004/agc-cred-2419caaf5f1f: secret "agc-cred-2419caaf5f1f" not found
Nov 02 13:13:08.929128 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:08.929115    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw: secret "agc-cred-2419caaf5f1f" not found
Nov 02 13:13:08.929293 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:08.929281    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials podName:c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:09.429261012 +0000 UTC m=+2735.535963855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-00d004-555947xbw" (UID: "c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063") : secret "agc-cred-2419caaf5f1f" not found
Nov 02 13:13:08.946346 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.946306    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:08.953648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:08.953620    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzjvn\" (UniqueName: \"kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-kube-api-access-zzjvn\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:09.087220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:09.087184    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:09.220220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:09.220121    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:09.350003 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:09.349958    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"]
Nov 02 13:13:09.441814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:09.441779    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:09.442055 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:09.441921    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-00d004/agc-cred-2419caaf5f1f: secret "agc-cred-2419caaf5f1f" not found
Nov 02 13:13:09.442055 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:09.441942    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw: secret "agc-cred-2419caaf5f1f" not found
Nov 02 13:13:09.442055 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:09.442026    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials podName:c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:10.442006425 +0000 UTC m=+2736.548709268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-00d004-555947xbw" (UID: "c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063") : secret "agc-cred-2419caaf5f1f" not found
Nov 02 13:13:09.448265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:09.448226    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:09.587797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:09.587713    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:10.144332 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:10.144299    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:10.453191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:10.453103    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"]
Nov 02 13:13:10.455619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:10.455586    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:10.455759 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:10.455728    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-00d004/agc-cred-2419caaf5f1f: secret "agc-cred-2419caaf5f1f" not found
Nov 02 13:13:10.455759 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:10.455744    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw: secret "agc-cred-2419caaf5f1f" not found
Nov 02 13:13:10.455854 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:10.455789    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials podName:c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:12.455774632 +0000 UTC m=+2738.562477474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-00d004-555947xbw" (UID: "c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063") : secret "agc-cred-2419caaf5f1f" not found
Nov 02 13:13:10.521324 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:10.521276    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:11.444611 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:11.444574    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"]
Nov 02 13:13:11.445081 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:11.445057    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw" podUID="c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063"
Nov 02 13:13:11.459832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:11.459795    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"]
Nov 02 13:13:11.545934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:11.545887    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:11.664278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:11.664233    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:11.980309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:11.980269    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:12.231398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.231310    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"
Nov 02 13:13:12.231398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.231381    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"
Nov 02 13:13:12.320577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.320548    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:12.331919 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.331892    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:12.384941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.384909    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-kube-api-access-zzjvn\" (UniqueName: \"kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-kube-api-access-zzjvn\") pod \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") "
Nov 02 13:13:12.385191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.385062    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-plugins-config-volume\") pod \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") "
Nov 02 13:13:12.385191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.385092    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-routing-overlay-volume\") pod \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") "
Nov 02 13:13:12.385487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.385463    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-routing-overlay-volume" pod "c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063" (UID: "c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:12.385551 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.385506    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-plugins-config-volume" pod "c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063" (UID: "c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:12.385811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.385796    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:12.385871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.385814    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:12.387371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.387348    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-kube-api-access-zzjvn" pod "c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063" (UID: "c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063"). InnerVolumeSpecName "kube-api-access-zzjvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:12.486906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.486808    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-00d004-555947xbw\" (UID: \"c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063\") " pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:12.487352 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:12.486975    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-00d004/agc-cred-2419caaf5f1f: secret "agc-cred-2419caaf5f1f" not found
Nov 02 13:13:12.487352 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:12.487011    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw: secret "agc-cred-2419caaf5f1f" not found
Nov 02 13:13:12.487352 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:12.487067    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials podName:c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063 nodeName:}" failed. No retries permitted until 2026-11-02 13:13:16.487052819 +0000 UTC m=+2742.593755671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-00d004-555947xbw" (UID: "c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063") : secret "agc-cred-2419caaf5f1f" not found
Nov 02 13:13:12.487352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:12.487200    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzjvn\" (UniqueName: \"kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-kube-api-access-zzjvn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:13.328578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:13.328536    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"
Nov 02 13:13:13.415206 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:13.415162    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"]
Nov 02 13:13:13.421251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:13.421213    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-555947xbw"]
Nov 02 13:13:13.503078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:13.503042    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:13.692362 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:13.692326    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:14.019270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.019192    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063" path="/var/lib/kubelet/pods/c2ebe8e9-2ce2-4a41-bd75-3c96b51ad063/volumes"
Nov 02 13:13:14.351028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.350911    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5" podUID="58a99f05-455d-42be-b61b-f5d7766c79f9" containerName="payload-processing" containerID="cri-o://1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d" gracePeriod=25
Nov 02 13:13:14.641962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.641934    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:13:14.721144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.721103    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-plugins-config-volume\") pod \"58a99f05-455d-42be-b61b-f5d7766c79f9\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") "
Nov 02 13:13:14.721373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.721165    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-kube-api-access-kfdbc\" (UniqueName: \"kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-kube-api-access-kfdbc\") pod \"58a99f05-455d-42be-b61b-f5d7766c79f9\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") "
Nov 02 13:13:14.721373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.721292    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-provider-credentials\") pod \"58a99f05-455d-42be-b61b-f5d7766c79f9\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") "
Nov 02 13:13:14.721502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.721410    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-routing-overlay-volume\") pod \"58a99f05-455d-42be-b61b-f5d7766c79f9\" (UID: \"58a99f05-455d-42be-b61b-f5d7766c79f9\") "
Nov 02 13:13:14.721794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.721629    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-plugins-config-volume" pod "58a99f05-455d-42be-b61b-f5d7766c79f9" (UID: "58a99f05-455d-42be-b61b-f5d7766c79f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:14.721946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.721824    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-routing-overlay-volume" pod "58a99f05-455d-42be-b61b-f5d7766c79f9" (UID: "58a99f05-455d-42be-b61b-f5d7766c79f9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:14.722705 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.722676    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:14.722817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.722709    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58a99f05-455d-42be-b61b-f5d7766c79f9-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:14.723971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.723946    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-provider-credentials" pod "58a99f05-455d-42be-b61b-f5d7766c79f9" (UID: "58a99f05-455d-42be-b61b-f5d7766c79f9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:14.724187 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.724167    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-kube-api-access-kfdbc" pod "58a99f05-455d-42be-b61b-f5d7766c79f9" (UID: "58a99f05-455d-42be-b61b-f5d7766c79f9"). InnerVolumeSpecName "kube-api-access-kfdbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:14.824026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.823974    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfdbc\" (UniqueName: \"kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-kube-api-access-kfdbc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:14.824026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:14.824028    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58a99f05-455d-42be-b61b-f5d7766c79f9-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:14.825033 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:14.824978    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:13:15.354587 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.354547    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:15.358330 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.358293    5879 generic.go:356] "Generic (PLEG): container finished" podID="58a99f05-455d-42be-b61b-f5d7766c79f9" containerID="1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d" exitCode=0
Nov 02 13:13:15.358504 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.358363    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"
Nov 02 13:13:15.358504 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.358380    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5" event={"ID":"58a99f05-455d-42be-b61b-f5d7766c79f9","Type":"ContainerDied","Data":"1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d"}
Nov 02 13:13:15.358504 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.358488    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5" event={"ID":"58a99f05-455d-42be-b61b-f5d7766c79f9","Type":"ContainerDied","Data":"4cbd164733c9a92c7fc8520f7bc8ace78b4cb80935fa39c16f13bdbb046e9f4b"}
Nov 02 13:13:15.358651 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.358517    5879 scope.go:122] "RemoveContainer" containerID="1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d"
Nov 02 13:13:15.384540 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.384508    5879 scope.go:122] "RemoveContainer" containerID="1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d"
Nov 02 13:13:15.385188 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:15.385074    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d\": container with ID starting with 1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d not found: ID does not exist" containerID="1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d"
Nov 02 13:13:15.385188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.385118    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d"} err="failed to get container status \"1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d\": rpc error: code = NotFound desc = could not find container \"1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d\": container with ID starting with 1e09014b92d99ec38f20089f130d179bd3036491d27528a8c59f12f3e1c84f2d not found: ID does not exist"
Nov 02 13:13:15.426566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.426530    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"]
Nov 02 13:13:15.439742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.439714    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-7d772dvg5"]
Nov 02 13:13:15.453679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.453642    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp" podUID="882fb9d3-1c94-4c23-a494-c9b4402d66ac" containerName="payload-processing" containerID="cri-o://9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b" gracePeriod=25
Nov 02 13:13:15.750341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.750310    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:15.755796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.755770    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:13:15.837650 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.837615    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-plugins-config-volume\") pod \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") "
Nov 02 13:13:15.837874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.837688    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-routing-overlay-volume\") pod \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") "
Nov 02 13:13:15.837874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.837771    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-kube-api-access-2cbfm\" (UniqueName: \"kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-kube-api-access-2cbfm\") pod \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") "
Nov 02 13:13:15.838041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.837901    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-provider-credentials\") pod \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\" (UID: \"882fb9d3-1c94-4c23-a494-c9b4402d66ac\") "
Nov 02 13:13:15.838109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.838064    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-plugins-config-volume" pod "882fb9d3-1c94-4c23-a494-c9b4402d66ac" (UID: "882fb9d3-1c94-4c23-a494-c9b4402d66ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:15.838419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.838399    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-routing-overlay-volume" pod "882fb9d3-1c94-4c23-a494-c9b4402d66ac" (UID: "882fb9d3-1c94-4c23-a494-c9b4402d66ac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:15.838549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.838533    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:15.838623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.838581    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/882fb9d3-1c94-4c23-a494-c9b4402d66ac-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:15.840855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.840822    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-provider-credentials" pod "882fb9d3-1c94-4c23-a494-c9b4402d66ac" (UID: "882fb9d3-1c94-4c23-a494-c9b4402d66ac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:15.841352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.841331    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-kube-api-access-2cbfm" pod "882fb9d3-1c94-4c23-a494-c9b4402d66ac" (UID: "882fb9d3-1c94-4c23-a494-c9b4402d66ac"). InnerVolumeSpecName "kube-api-access-2cbfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:15.940535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.940450    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:15.940535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.940480    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cbfm\" (UniqueName: \"kubernetes.io/projected/882fb9d3-1c94-4c23-a494-c9b4402d66ac-kube-api-access-2cbfm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:15.958832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:15.958799    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:16.019186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.019149    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a99f05-455d-42be-b61b-f5d7766c79f9" path="/var/lib/kubelet/pods/58a99f05-455d-42be-b61b-f5d7766c79f9/volumes"
Nov 02 13:13:16.086952 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.086902    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:16.227146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.227056    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:16.370597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.370552    5879 generic.go:356] "Generic (PLEG): container finished" podID="882fb9d3-1c94-4c23-a494-c9b4402d66ac" containerID="9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b" exitCode=0
Nov 02 13:13:16.370818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.370598    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp" event={"ID":"882fb9d3-1c94-4c23-a494-c9b4402d66ac","Type":"ContainerDied","Data":"9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b"}
Nov 02 13:13:16.370818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.370674    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp" event={"ID":"882fb9d3-1c94-4c23-a494-c9b4402d66ac","Type":"ContainerDied","Data":"0043b4436ba4c7fc00345c1df4f074d823135d39ff50677702e07dc23f93fb70"}
Nov 02 13:13:16.370818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.370712    5879 scope.go:122] "RemoveContainer" containerID="9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b"
Nov 02 13:13:16.371135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.371110    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"
Nov 02 13:13:16.383663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.383640    5879 scope.go:122] "RemoveContainer" containerID="9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b"
Nov 02 13:13:16.384044 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:16.383984    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b\": container with ID starting with 9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b not found: ID does not exist" containerID="9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b"
Nov 02 13:13:16.384125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.384055    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b"} err="failed to get container status \"9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b\": rpc error: code = NotFound desc = could not find container \"9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b\": container with ID starting with 9801e522e1a970984a2d4683cd40e3963040af6e0aeea6a5c9b15fb8290ee51b not found: ID does not exist"
Nov 02 13:13:16.430064 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.430019    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"]
Nov 02 13:13:16.444298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.444263    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-035dac/payload-processing-external-model-e2e-extmod-a-035dac-7577497xp"]
Nov 02 13:13:16.460442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.460395    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7" podUID="97b92183-a743-4f2a-be41-4faa2fc5cbe4" containerName="payload-processing" containerID="cri-o://5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03" gracePeriod=25
Nov 02 13:13:16.608394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.608362    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:16.743059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.742757    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:16.771490 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.771412    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:13:16.853201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.853161    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-provider-credentials\") pod \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") "
Nov 02 13:13:16.853408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.853368    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-routing-overlay-volume\") pod \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") "
Nov 02 13:13:16.853524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.853508    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-plugins-config-volume\") pod \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") "
Nov 02 13:13:16.853675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.853653    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-kube-api-access-nchqt\" (UniqueName: \"kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-kube-api-access-nchqt\") pod \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\" (UID: \"97b92183-a743-4f2a-be41-4faa2fc5cbe4\") "
Nov 02 13:13:16.853902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.853868    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-routing-overlay-volume" pod "97b92183-a743-4f2a-be41-4faa2fc5cbe4" (UID: "97b92183-a743-4f2a-be41-4faa2fc5cbe4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:16.854056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.854020    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-plugins-config-volume" pod "97b92183-a743-4f2a-be41-4faa2fc5cbe4" (UID: "97b92183-a743-4f2a-be41-4faa2fc5cbe4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:13:16.854437 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.854365    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:16.854437 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.854389    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b92183-a743-4f2a-be41-4faa2fc5cbe4-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:16.856021 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.855969    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-provider-credentials" pod "97b92183-a743-4f2a-be41-4faa2fc5cbe4" (UID: "97b92183-a743-4f2a-be41-4faa2fc5cbe4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:16.856575 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.856555    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-kube-api-access-nchqt" pod "97b92183-a743-4f2a-be41-4faa2fc5cbe4" (UID: "97b92183-a743-4f2a-be41-4faa2fc5cbe4"). InnerVolumeSpecName "kube-api-access-nchqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:13:16.956220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.956189    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nchqt\" (UniqueName: \"kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-kube-api-access-nchqt\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:16.956220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:16.956214    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b92183-a743-4f2a-be41-4faa2fc5cbe4-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:13:17.189756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:17.189652    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:17.331908 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:17.331874    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:17.386375 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:17.386334    5879 generic.go:356] "Generic (PLEG): container finished" podID="97b92183-a743-4f2a-be41-4faa2fc5cbe4" containerID="5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03" exitCode=0
Nov 02 13:13:17.386375 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:17.386379    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"
Nov 02 13:13:17.386649 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:17.386406    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7" event={"ID":"97b92183-a743-4f2a-be41-4faa2fc5cbe4","Type":"ContainerDied","Data":"5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03"}
Nov 02 13:13:17.386649 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:17.386463    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7" event={"ID":"97b92183-a743-4f2a-be41-4faa2fc5cbe4","Type":"ContainerDied","Data":"5464359a071d6ee7b11a7085f7187640bdc37975e1ef4f3f7c4ec731128e20db"}
Nov 02 13:13:17.386649 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:17.386489    5879 scope.go:122] "RemoveContainer" containerID="5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03"
Nov 02 13:13:17.401346 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:17.401321    5879 scope.go:122] "RemoveContainer" containerID="5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03"
Nov 02 13:13:17.401724 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:17.401691    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03\": container with ID starting with 5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03 not found: ID does not exist" containerID="5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03"
Nov 02 13:13:17.401808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:17.401724    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03"} err="failed to get container status \"5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03\": rpc error: code = NotFound desc = could not find container \"5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03\": container with ID starting with 5bb136af44c29a8fb05cf5985b0b5cf61a9884ae0dbe6ced415ef8a6730a4a03 not found: ID does not exist"
Nov 02 13:13:17.469070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:17.463710    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"]
Nov 02 13:13:17.490340 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:17.490301    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-00d004/payload-processing-external-model-e2e-extmod-b-00d004-78d9n72w7"]
Nov 02 13:13:17.954026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:17.952234    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:18.021746 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:18.021709    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="882fb9d3-1c94-4c23-a494-c9b4402d66ac" path="/var/lib/kubelet/pods/882fb9d3-1c94-4c23-a494-c9b4402d66ac/volumes"
Nov 02 13:13:18.022438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:18.022414    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b92183-a743-4f2a-be41-4faa2fc5cbe4" path="/var/lib/kubelet/pods/97b92183-a743-4f2a-be41-4faa2fc5cbe4/volumes"
Nov 02 13:13:18.864070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:18.859584    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:19.232048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.231896    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:19.398228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.398194    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:19.467568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.467528    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"]
Nov 02 13:13:19.469036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.469005    5879 cpu_manager.go:414] "containerMap: removing container" podUID="97b92183-a743-4f2a-be41-4faa2fc5cbe4" containerName="payload-processing"
Nov 02 13:13:19.469234 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.469217    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b92183-a743-4f2a-be41-4faa2fc5cbe4" containerName="payload-processing"
Nov 02 13:13:19.469418 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.469403    5879 cpu_manager.go:414] "containerMap: removing container" podUID="882fb9d3-1c94-4c23-a494-c9b4402d66ac" containerName="payload-processing"
Nov 02 13:13:19.469518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.469505    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882fb9d3-1c94-4c23-a494-c9b4402d66ac" containerName="payload-processing"
Nov 02 13:13:19.469630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.469618    5879 cpu_manager.go:414] "containerMap: removing container" podUID="58a99f05-455d-42be-b61b-f5d7766c79f9" containerName="payload-processing"
Nov 02 13:13:19.469765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.469750    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a99f05-455d-42be-b61b-f5d7766c79f9" containerName="payload-processing"
Nov 02 13:13:19.474153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.474129    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="882fb9d3-1c94-4c23-a494-c9b4402d66ac" containerName="payload-processing"
Nov 02 13:13:19.474336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.474321    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a99f05-455d-42be-b61b-f5d7766c79f9" containerName="payload-processing"
Nov 02 13:13:19.474457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.474444    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b92183-a743-4f2a-be41-4faa2fc5cbe4" containerName="payload-processing"
Nov 02 13:13:19.483888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.483812    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.509941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.508402    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3396f\"/\"openshift-service-ca.crt\""
Nov 02 13:13:19.509941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.509202    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3396f\"/\"kube-root-ca.crt\""
Nov 02 13:13:19.509941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.509281    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3396f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e3396f\""
Nov 02 13:13:19.509941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.509360    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3396f\"/\"routing-overlay\""
Nov 02 13:13:19.517343 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.515219    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"]
Nov 02 13:13:19.555980 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.555943    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3396f\"/\"agc-cred-be7d57325b63\""
Nov 02 13:13:19.658091 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.557585    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3396f\"/\"payload-processing-external-model-e2e--528e896e-dockercfg-k895v\""
Nov 02 13:13:19.685434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.685392    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnqtp\" (UniqueName: \"kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-kube-api-access-gnqtp\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.685762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.685733    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.686088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.686065    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.686262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.686243    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.792458 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.792351    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.793725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.793699    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.795948 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.795925    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnqtp\" (UniqueName: \"kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-kube-api-access-gnqtp\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.796500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.796478    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.801233 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.793638    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.801702 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.801638    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.830636 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.830603    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.861407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.841324    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnqtp\" (UniqueName: \"kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-kube-api-access-gnqtp\") pod \"payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") " pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:19.861407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:19.859290    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:20.582569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:20.582457    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:20.611542 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:20.610902    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"]
Nov 02 13:13:21.172505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.172410    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"]
Nov 02 13:13:21.180431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.180396    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.215151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.214580    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"routing-overlay\""
Nov 02 13:13:21.216916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.215775    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"openshift-service-ca.crt\""
Nov 02 13:13:21.216916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.215862    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"agc-cred-321c776f703e\""
Nov 02 13:13:21.216916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.216036    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"kube-root-ca.crt\""
Nov 02 13:13:21.219228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.219203    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"]
Nov 02 13:13:21.233049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.224199    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-14fcb6\""
Nov 02 13:13:21.233049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.224281    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-14fcb6\"/\"payload-processing-external-model-e2e--ee70ab17-dockercfg-v9ths\""
Nov 02 13:13:21.311902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.307367    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72gnq\" (UniqueName: \"kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-kube-api-access-72gnq\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.311902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.307604    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.311902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.307689    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.311902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.307726    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.415019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.413557    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.415019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.413623    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.415019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.413737    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72gnq\" (UniqueName: \"kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-kube-api-access-72gnq\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.415019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.414008    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.415019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.414945    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.425616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.417580    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.425616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.423142    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.433044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.430177    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:21.449044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.448958    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72gnq\" (UniqueName: \"kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-kube-api-access-72gnq\") pod \"payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") " pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.507244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.505977    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq" event={"ID":"4c2e5684-7388-43fd-86fd-044b4e9dd54a","Type":"ContainerStarted","Data":"97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18"}
Nov 02 13:13:21.507244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.506102    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq" event={"ID":"4c2e5684-7388-43fd-86fd-044b4e9dd54a","Type":"ContainerStarted","Data":"cb2ad40c8e74719be59902deeef7c36591932ff76bbbb766cd7279afe1aa1ebd"}
Nov 02 13:13:21.507244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.507176    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:21.517022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.516726    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:21.574286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.574220    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq" podStartSLOduration=2.574200066 podStartE2EDuration="2.574200066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:21.570796124 +0000 UTC m=+2747.677498975" watchObservedRunningTime="2026-11-02 13:13:21.574200066 +0000 UTC m=+2747.680902918"
Nov 02 13:13:21.916258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:21.916220    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"]
Nov 02 13:13:21.922220 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:13:21.922190    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aeb79cf_cecb_4c34_8fb3_f75dc5dc86f8.slice/crio-31a365023d176f5a0e2dc9f454b1c8ee08de01eab29627614aefb56b4fe6e6d4 WatchSource:0}: Error finding container 31a365023d176f5a0e2dc9f454b1c8ee08de01eab29627614aefb56b4fe6e6d4: Status 404 returned error can't find the container with id 31a365023d176f5a0e2dc9f454b1c8ee08de01eab29627614aefb56b4fe6e6d4
Nov 02 13:13:22.516753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:22.516712    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q" event={"ID":"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8","Type":"ContainerStarted","Data":"7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f"}
Nov 02 13:13:22.516954 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:22.516783    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q" event={"ID":"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8","Type":"ContainerStarted","Data":"31a365023d176f5a0e2dc9f454b1c8ee08de01eab29627614aefb56b4fe6e6d4"}
Nov 02 13:13:22.516954 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:22.516814    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:22.564736 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:22.564673    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q" podStartSLOduration=1.5646547499999999 podStartE2EDuration="1.56465475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:22.561281836 +0000 UTC m=+2748.667984689" watchObservedRunningTime="2026-11-02 13:13:22.56465475 +0000 UTC m=+2748.671357600"
Nov 02 13:13:23.126158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:23.126121    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:23.266964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:23.266933    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:23.424805 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:23.424672    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:24.934785 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:24.934742    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:25.062513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.062476    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:25.223071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.218055    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"]
Nov 02 13:13:25.277473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.276232    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"]
Nov 02 13:13:25.277473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.276363    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"
Nov 02 13:13:25.292348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.292297    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:25.296477 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.296446    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5f8899-dockercfg-72pwx\""
Nov 02 13:13:25.298038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.297364    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5f8899\""
Nov 02 13:13:25.392683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.392648    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f00ea282-bd5f-4e08-8def-89fb409a4d24-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68\" (UID: \"f00ea282-bd5f-4e08-8def-89fb409a4d24\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"
Nov 02 13:13:25.392917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.392848    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjqlx\" (UniqueName: \"kubernetes.io/projected/f00ea282-bd5f-4e08-8def-89fb409a4d24-kube-api-access-gjqlx\") pod \"payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68\" (UID: \"f00ea282-bd5f-4e08-8def-89fb409a4d24\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"
Nov 02 13:13:25.493806 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.493721    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjqlx\" (UniqueName: \"kubernetes.io/projected/f00ea282-bd5f-4e08-8def-89fb409a4d24-kube-api-access-gjqlx\") pod \"payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68\" (UID: \"f00ea282-bd5f-4e08-8def-89fb409a4d24\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"
Nov 02 13:13:25.493964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.493934    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f00ea282-bd5f-4e08-8def-89fb409a4d24-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68\" (UID: \"f00ea282-bd5f-4e08-8def-89fb409a4d24\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"
Nov 02 13:13:25.494651 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.494633    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f00ea282-bd5f-4e08-8def-89fb409a4d24-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68\" (UID: \"f00ea282-bd5f-4e08-8def-89fb409a4d24\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"
Nov 02 13:13:25.519584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.519554    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjqlx\" (UniqueName: \"kubernetes.io/projected/f00ea282-bd5f-4e08-8def-89fb409a4d24-kube-api-access-gjqlx\") pod \"payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68\" (UID: \"f00ea282-bd5f-4e08-8def-89fb409a4d24\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"
Nov 02 13:13:25.618938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.618897    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"
Nov 02 13:13:25.935152 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:13:25.935111    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00ea282_bd5f_4e08_8def_89fb409a4d24.slice/crio-53712ad8f066a90bc2535229943d22758f3f72784e2d79e183080dc1e67dc230 WatchSource:0}: Error finding container 53712ad8f066a90bc2535229943d22758f3f72784e2d79e183080dc1e67dc230: Status 404 returned error can't find the container with id 53712ad8f066a90bc2535229943d22758f3f72784e2d79e183080dc1e67dc230
Nov 02 13:13:25.937728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:25.937699    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"]
Nov 02 13:13:26.555565 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:26.555526    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68" event={"ID":"f00ea282-bd5f-4e08-8def-89fb409a4d24","Type":"ContainerStarted","Data":"fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe"}
Nov 02 13:13:26.555791 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:26.555592    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"
Nov 02 13:13:26.555791 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:26.555608    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68" event={"ID":"f00ea282-bd5f-4e08-8def-89fb409a4d24","Type":"ContainerStarted","Data":"53712ad8f066a90bc2535229943d22758f3f72784e2d79e183080dc1e67dc230"}
Nov 02 13:13:26.603135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:26.603080    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68" podStartSLOduration=1.60306254 podStartE2EDuration="1.60306254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:26.598826126 +0000 UTC m=+2752.705528977" watchObservedRunningTime="2026-11-02 13:13:26.60306254 +0000 UTC m=+2752.709765391"
Nov 02 13:13:26.938110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:26.938076    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:28.566118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:28.566074    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:30.339830 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:30.339796    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:31.888030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:31.887948    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:32.694075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.694032    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"]
Nov 02 13:13:32.700604 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.700574    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.723326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.723291    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"]
Nov 02 13:13:32.731188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.731155    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-90cda9-openshift-default-dockercfg-xm76t\""
Nov 02 13:13:32.808407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.808362    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbtb4\" (UniqueName: \"kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-kube-api-access-pbtb4\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.808407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.808401    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-token\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.808633 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.808533    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-data\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.808633 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.808569    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-podinfo\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.808712 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.808662    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-credential-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.808712 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.808685    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-certs\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.808779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.808710    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.808817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.808801    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-envoy\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.808853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.808840    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf891163-0b6a-4951-8730-04154ea0b7e9-istiod-ca-cert\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.910318 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.910281    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-envoy\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.910891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.910341    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf891163-0b6a-4951-8730-04154ea0b7e9-istiod-ca-cert\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.910891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.910492    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbtb4\" (UniqueName: \"kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-kube-api-access-pbtb4\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.910891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.910516    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-token\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.910891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.910587    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-data\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.910891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.910619    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-podinfo\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.911218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.910947    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-credential-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.911218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.910981    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-certs\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.911218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.911047    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.911218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.911092    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-data\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.911625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.911292    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-credential-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.911625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.911418    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-certs\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.911625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.911474    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-socket\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.912134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.912108    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf891163-0b6a-4951-8730-04154ea0b7e9-istiod-ca-cert\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.913323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.913288    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-envoy\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.913595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.913578    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-podinfo\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.937231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.937195    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-token\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:32.937419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:32.937299    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbtb4\" (UniqueName: \"kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-kube-api-access-pbtb4\") pod \"e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") " pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:33.018850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:33.018759    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:33.369790 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:13:33.369758    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf891163_0b6a_4951_8730_04154ea0b7e9.slice/crio-5dc0cca9da57c6c733d2ec676ce0b2f4dce6fe8b2788426a060d8a081e2e87bc WatchSource:0}: Error finding container 5dc0cca9da57c6c733d2ec676ce0b2f4dce6fe8b2788426a060d8a081e2e87bc: Status 404 returned error can't find the container with id 5dc0cca9da57c6c733d2ec676ce0b2f4dce6fe8b2788426a060d8a081e2e87bc
Nov 02 13:13:33.370939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:33.370914    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"]
Nov 02 13:13:33.373423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:33.373380    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:13:33.373538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:33.373470    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:13:33.373538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:33.373520    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:13:33.445013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:33.444955    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:33.532787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:33.532756    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:13:33.532961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:33.532835    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:13:33.625629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:33.625513    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp" event={"ID":"cf891163-0b6a-4951-8730-04154ea0b7e9","Type":"ContainerStarted","Data":"a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a"}
Nov 02 13:13:33.625629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:33.625598    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp" event={"ID":"cf891163-0b6a-4951-8730-04154ea0b7e9","Type":"ContainerStarted","Data":"5dc0cca9da57c6c733d2ec676ce0b2f4dce6fe8b2788426a060d8a081e2e87bc"}
Nov 02 13:13:33.676678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:33.676588    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp" podStartSLOduration=1.676572636 podStartE2EDuration="1.676572636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:13:33.673301769 +0000 UTC m=+2759.780004621" watchObservedRunningTime="2026-11-02 13:13:33.676572636 +0000 UTC m=+2759.783275486"
Nov 02 13:13:34.019413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:34.019328    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:34.021245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:34.021220    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body=
Nov 02 13:13:34.021340 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:34.021272    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp" podUID="cf891163-0b6a-4951-8730-04154ea0b7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused"
Nov 02 13:13:35.019761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:35.019731    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body=
Nov 02 13:13:35.020202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:35.019779    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp" podUID="cf891163-0b6a-4951-8730-04154ea0b7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused"
Nov 02 13:13:35.172214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:35.172177    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:36.019896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:36.019864    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body=
Nov 02 13:13:36.020356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:36.019920    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp" podUID="cf891163-0b6a-4951-8730-04154ea0b7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused"
Nov 02 13:13:36.678846 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:36.678811    5879 scope.go:122] "RemoveContainer" containerID="a7efb6b87f838b30e28d5be0e14bea184458c846933b49dc3a1144b750ed1df4"
Nov 02 13:13:36.713108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:36.713084    5879 scope.go:122] "RemoveContainer" containerID="d28db5c1a1d9a22c1582778200367191532fb8e796c4c225820b45f0fa02a274"
Nov 02 13:13:36.855096 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:36.855067    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:37.019614 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:37.019507    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body=
Nov 02 13:13:37.019614 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:37.019582    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp" podUID="cf891163-0b6a-4951-8730-04154ea0b7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused"
Nov 02 13:13:37.568427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:37.568397    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"
Nov 02 13:13:38.019135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:38.019100    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body=
Nov 02 13:13:38.019409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:38.019157    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp" podUID="cf891163-0b6a-4951-8730-04154ea0b7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused"
Nov 02 13:13:38.032536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:38.032500    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"]
Nov 02 13:13:38.036562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:38.036533    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c20986d2-4qwn6"]
Nov 02 13:13:38.549644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:38.549611    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:39.019982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:39.019945    5879 patch_prober.go:29] interesting pod/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused" start-of-body=
Nov 02 13:13:39.020437 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:39.020024    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp" podUID="cf891163-0b6a-4951-8730-04154ea0b7e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.39:15021/healthz/ready\": dial tcp 10.134.1.39:15021: connect: connection refused"
Nov 02 13:13:39.050711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:39.050663    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:39.159908 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:39.159871    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:40.018956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:40.018921    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e7ad32-cade-4621-846f-d1a057f39c47" path="/var/lib/kubelet/pods/f2e7ad32-cade-4621-846f-d1a057f39c47/volumes"
Nov 02 13:13:40.026586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:40.026550    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:40.027774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:40.027742    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:40.028867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:40.028842    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:13:40.165841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:40.165801    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:41.710770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:41.710730    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:41.839355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:41.839317    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:41.969727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:41.969641    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:43.449926 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:43.449893    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:44.823889 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:13:44.823849    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:13:44.955507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:44.955462    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:45.085176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:45.085087    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:45.368295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:45.368263    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:45.492506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:45.492471    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:46.491479 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:46.491443    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:48.026223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:48.026187    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:48.148204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:48.148175    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:48.303897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:48.303794    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:49.787435 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:49.787402    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:51.032722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:51.032679    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"]
Nov 02 13:13:51.046600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:51.046567    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd569777-fmgqq"]
Nov 02 13:13:51.476351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:51.476316    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:52.019595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:52.019559    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e7e43d-334c-4367-8642-5669d5df4c8e" path="/var/lib/kubelet/pods/e0e7e43d-334c-4367-8642-5669d5df4c8e/volumes"
Nov 02 13:13:53.122280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:53.122246    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:54.858612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:54.858578    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:56.391591 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:56.391555    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:57.964598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:57.964567    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:13:59.791713 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:13:59.791669    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:01.466120 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:01.466066    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:03.034865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:03.034828    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"]
Nov 02 13:14:03.049354 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:03.049324    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08689e60-6vr6m"]
Nov 02 13:14:03.199839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:03.199804    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:04.021119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:04.021084    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be954e1c-33e9-44cf-9365-c88f37d39856" path="/var/lib/kubelet/pods/be954e1c-33e9-44cf-9365-c88f37d39856/volumes"
Nov 02 13:14:04.040084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:04.040053    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"]
Nov 02 13:14:04.053165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:04.053136    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59beaad8-vrlf2"]
Nov 02 13:14:04.820291 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:04.820258    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:06.019969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:06.019933    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa80df6-2b05-4ff5-b27b-0cf4a8958848" path="/var/lib/kubelet/pods/aaa80df6-2b05-4ff5-b27b-0cf4a8958848/volumes"
Nov 02 13:14:06.031471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:06.031439    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"]
Nov 02 13:14:06.045547 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:06.045511    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2973c15c-5ktrt"]
Nov 02 13:14:06.472562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:06.472531    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:08.018650 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:08.018615    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caff3b86-d5ba-40bf-a7d2-9192f4d509d5" path="/var/lib/kubelet/pods/caff3b86-d5ba-40bf-a7d2-9192f4d509d5/volumes"
Nov 02 13:14:08.123444 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:08.123393    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:08.259276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:08.259232    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:08.370443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:08.370410    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:09.773016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:09.772958    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:11.347810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:11.347772    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:12.952537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:12.952499    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:14.032729 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:14.032653    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"]
Nov 02 13:14:14.046348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:14.046314    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a11a10-gxjgk"]
Nov 02 13:14:14.535717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:14.535677    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:14.823215 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:14:14.823125    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:14:15.038638 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:15.038606    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"]
Nov 02 13:14:15.052973 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:15.052933    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-152184f0-d9pkf"]
Nov 02 13:14:16.021177 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:16.021141    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056e1508-5e60-4712-8765-e871d3339852" path="/var/lib/kubelet/pods/056e1508-5e60-4712-8765-e871d3339852/volumes"
Nov 02 13:14:16.021717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:16.021701    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d2cd78d-8135-471c-b1de-176466518666" path="/var/lib/kubelet/pods/5d2cd78d-8135-471c-b1de-176466518666/volumes"
Nov 02 13:14:16.154653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:16.154620    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:17.757415 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:17.757385    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:19.444719 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:19.444687    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:21.042917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:21.042882    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:22.680646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:22.680575    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:24.342152 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:24.342111    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:25.879731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:25.879676    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:27.033606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:27.033566    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"]
Nov 02 13:14:27.047595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:27.047554    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edabc533-jpv77"]
Nov 02 13:14:27.463345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:27.463298    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:28.018795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:28.018759    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5dbb607-175b-4837-bb94-6fb2710b5dea" path="/var/lib/kubelet/pods/b5dbb607-175b-4837-bb94-6fb2710b5dea/volumes"
Nov 02 13:14:29.139512 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:29.139454    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:30.871776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:30.871735    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:32.437663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:32.437606    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:34.095104 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:34.095067    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:34.259749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:34.259705    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:35.898419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:35.898379    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:36.764677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:36.764642    5879 scope.go:122] "RemoveContainer" containerID="48bbf0bcd68e1e1f150de4fff1d0e7aef6182a43d6aa72afc1af38bb31086d43"
Nov 02 13:14:36.777924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:36.777896    5879 scope.go:122] "RemoveContainer" containerID="d55e0adcd808b9319d5356d9ad82a514a66c72616b61e8ea3301f856d10e8d06"
Nov 02 13:14:36.789101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:36.789068    5879 scope.go:122] "RemoveContainer" containerID="1f47390f1dd9f7e0f1ebdbb4c599de0b1aca24ebc27649570ed3ef828c9fe841"
Nov 02 13:14:37.013411 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.013375    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"]
Nov 02 13:14:37.020263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.020192    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.058897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.058865    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bbb7d0\"/\"default-dockercfg-w78g6\""
Nov 02 13:14:37.059113 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.059064    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bbb7d0\"/\"test-model-bbb7d0-kserve-self-signed-certs\""
Nov 02 13:14:37.082024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.075403    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bbb7d0\"/\"openshift-service-ca.crt\""
Nov 02 13:14:37.082024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.075403    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bbb7d0\"/\"kube-root-ca.crt\""
Nov 02 13:14:37.082024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.076918    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:37.082363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.082102    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"]
Nov 02 13:14:37.129209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.129167    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-model-cache\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.129406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.129241    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-tmp-dir\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.129406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.129327    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnvzx\" (UniqueName: \"kubernetes.io/projected/0f015abe-55db-486b-a074-43553ea9e791-kube-api-access-hnvzx\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.129406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.129365    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f015abe-55db-486b-a074-43553ea9e791-tls-certs\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.129521 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.129470    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-home\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.129591 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.129569    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-dshm\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.230707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.230664    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-tmp-dir\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.230925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.230722    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnvzx\" (UniqueName: \"kubernetes.io/projected/0f015abe-55db-486b-a074-43553ea9e791-kube-api-access-hnvzx\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.230925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.230752    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f015abe-55db-486b-a074-43553ea9e791-tls-certs\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.230925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.230803    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-home\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.231194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.230967    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-dshm\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.231194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.231143    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-model-cache\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.231306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.231184    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-tmp-dir\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.231306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.231250    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-home\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.231463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.231442    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-model-cache\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.233747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.233720    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-dshm\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.233868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.233852    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f015abe-55db-486b-a074-43553ea9e791-tls-certs\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.324148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.305803    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnvzx\" (UniqueName: \"kubernetes.io/projected/0f015abe-55db-486b-a074-43553ea9e791-kube-api-access-hnvzx\") pod \"test-model-bbb7d0-kserve-59c45f57df-bkrgg\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") " pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.343038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.342984    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:37.678196 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:14:37.678152    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f015abe_55db_486b_a074_43553ea9e791.slice/crio-9c3b333b921f6f9605237a560cbcedecc84a7c12bca40df7e5beae12ba7a22f6 WatchSource:0}: Error finding container 9c3b333b921f6f9605237a560cbcedecc84a7c12bca40df7e5beae12ba7a22f6: Status 404 returned error can't find the container with id 9c3b333b921f6f9605237a560cbcedecc84a7c12bca40df7e5beae12ba7a22f6
Nov 02 13:14:37.679606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.679579    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"]
Nov 02 13:14:37.840330 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:37.840288    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:38.305795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:38.305754    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg" event={"ID":"0f015abe-55db-486b-a074-43553ea9e791","Type":"ContainerStarted","Data":"18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4"}
Nov 02 13:14:38.306325 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:38.305806    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg" event={"ID":"0f015abe-55db-486b-a074-43553ea9e791","Type":"ContainerStarted","Data":"9c3b333b921f6f9605237a560cbcedecc84a7c12bca40df7e5beae12ba7a22f6"}
Nov 02 13:14:38.343325 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:38.343275    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:38.364672 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:38.364613    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg" podStartSLOduration=2.364597899 podStartE2EDuration="2.364597899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:14:38.361824355 +0000 UTC m=+2824.468527205" watchObservedRunningTime="2026-11-02 13:14:38.364597899 +0000 UTC m=+2824.471300827"
Nov 02 13:14:39.188690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:39.188656    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:39.615319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:39.613339    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:41.118203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:41.118172    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:41.363211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:41.363173    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:41.618845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:41.618807    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:42.047291 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:42.047158    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:42.167571 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:42.167515    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:43.056802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:43.056743    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:44.820313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:44.820272    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:44.825820 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:14:44.825784    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:14:45.036605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:45.036569    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"]
Nov 02 13:14:45.049798 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:45.049766    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513775d4-5j8p6"]
Nov 02 13:14:46.019966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:46.019932    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c984db05-067d-4977-be7d-da62869b388f" path="/var/lib/kubelet/pods/c984db05-067d-4977-be7d-da62869b388f/volumes"
Nov 02 13:14:46.154427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:46.154389    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:46.296718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:46.296636    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:47.150775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:47.150732    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:47.344017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:47.343952    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:47.357787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:47.357757    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:47.400331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:47.400299    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:14:48.069838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.069804    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:48.219566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.219532    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:48.247338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.247300    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"]
Nov 02 13:14:48.253545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.253505    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.256530 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.256501    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"]
Nov 02 13:14:48.261810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.261772    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"
Nov 02 13:14:48.286880 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.286846    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-361c3a\""
Nov 02 13:14:48.291635 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.291583    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"]
Nov 02 13:14:48.324685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.324053    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.325057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.325022    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.325212 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.325087    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.325212 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.325112    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.325212 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.325170    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.325212 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.325196    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.325470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.325349    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.325524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.325465    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.325524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.325508    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh27b\" (UniqueName: \"kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-kube-api-access-vh27b\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.328292 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.328031    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"]
Nov 02 13:14:48.380019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.379149    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-361c3a-dockercfg-gqqmv\""
Nov 02 13:14:48.395028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.381256    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5f8899-openshift-default-dockercfg-rrzwf\""
Nov 02 13:14:48.445045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.429398    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.445045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.429577    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.445045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.429707    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.445045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.429752    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3169f74e-ae11-4f79-80bd-5e400884ac03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn\" (UID: \"3169f74e-ae11-4f79-80bd-5e400884ac03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"
Nov 02 13:14:48.445045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.429789    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh27b\" (UniqueName: \"kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-kube-api-access-vh27b\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.445045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.429869    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.445045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.429922    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rsjq\" (UniqueName: \"kubernetes.io/projected/3169f74e-ae11-4f79-80bd-5e400884ac03-kube-api-access-6rsjq\") pod \"payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn\" (UID: \"3169f74e-ae11-4f79-80bd-5e400884ac03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"
Nov 02 13:14:48.445045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.429964    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.462020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.453130    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.462020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.453179    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.462020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.453266    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.462020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.454057    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-credential-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.462020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.454514    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-socket\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.462020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.456189    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istiod-ca-cert\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.462020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.458612    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-data\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.462020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.459527    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-certs\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.491017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.464440    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-envoy\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.503108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.503054    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-podinfo\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.537117 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.537083    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-token\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.540047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.539978    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh27b\" (UniqueName: \"kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-kube-api-access-vh27b\") pod \"e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") " pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.555353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.555320    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3169f74e-ae11-4f79-80bd-5e400884ac03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn\" (UID: \"3169f74e-ae11-4f79-80bd-5e400884ac03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"
Nov 02 13:14:48.555541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.555409    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsjq\" (UniqueName: \"kubernetes.io/projected/3169f74e-ae11-4f79-80bd-5e400884ac03-kube-api-access-6rsjq\") pod \"payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn\" (UID: \"3169f74e-ae11-4f79-80bd-5e400884ac03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"
Nov 02 13:14:48.556201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.556177    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3169f74e-ae11-4f79-80bd-5e400884ac03-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn\" (UID: \"3169f74e-ae11-4f79-80bd-5e400884ac03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"
Nov 02 13:14:48.575653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.575577    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:48.581761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.581732    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rsjq\" (UniqueName: \"kubernetes.io/projected/3169f74e-ae11-4f79-80bd-5e400884ac03-kube-api-access-6rsjq\") pod \"payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn\" (UID: \"3169f74e-ae11-4f79-80bd-5e400884ac03\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"
Nov 02 13:14:48.584563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.584535    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"
Nov 02 13:14:48.933366 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:14:48.933324    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3169f74e_ae11_4f79_80bd_5e400884ac03.slice/crio-74dfd75792b60e70e352b7bedcd9bd465f34c6c5d40c80d7c0b32d87b866b423 WatchSource:0}: Error finding container 74dfd75792b60e70e352b7bedcd9bd465f34c6c5d40c80d7c0b32d87b866b423: Status 404 returned error can't find the container with id 74dfd75792b60e70e352b7bedcd9bd465f34c6c5d40c80d7c0b32d87b866b423
Nov 02 13:14:48.934884 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:14:48.934847    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77bd7e8_4b52_427b_90c5_ad7460330a8a.slice/crio-3da82186d464367370c8382298e80f7ff8e58c5db6ffa358a859470c503229ee WatchSource:0}: Error finding container 3da82186d464367370c8382298e80f7ff8e58c5db6ffa358a859470c503229ee: Status 404 returned error can't find the container with id 3da82186d464367370c8382298e80f7ff8e58c5db6ffa358a859470c503229ee
Nov 02 13:14:48.935198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.935175    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"]
Nov 02 13:14:48.938359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.938316    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:14:48.938499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.938414    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:14:48.938499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.938473    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:14:48.940822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:48.940802    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"]
Nov 02 13:14:49.176400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:49.176364    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:49.461321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:49.461229    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn" event={"ID":"3169f74e-ae11-4f79-80bd-5e400884ac03","Type":"ContainerStarted","Data":"0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc"}
Nov 02 13:14:49.461321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:49.461290    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn" event={"ID":"3169f74e-ae11-4f79-80bd-5e400884ac03","Type":"ContainerStarted","Data":"74dfd75792b60e70e352b7bedcd9bd465f34c6c5d40c80d7c0b32d87b866b423"}
Nov 02 13:14:49.461896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:49.461386    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"
Nov 02 13:14:49.463459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:49.463431    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm" event={"ID":"c77bd7e8-4b52-427b-90c5-ad7460330a8a","Type":"ContainerStarted","Data":"77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369"}
Nov 02 13:14:49.463604 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:49.463487    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm" event={"ID":"c77bd7e8-4b52-427b-90c5-ad7460330a8a","Type":"ContainerStarted","Data":"3da82186d464367370c8382298e80f7ff8e58c5db6ffa358a859470c503229ee"}
Nov 02 13:14:49.506572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:49.506501    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn" podStartSLOduration=1.506480018 podStartE2EDuration="1.506480018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:14:49.505194834 +0000 UTC m=+2835.611897686" watchObservedRunningTime="2026-11-02 13:14:49.506480018 +0000 UTC m=+2835.613182870"
Nov 02 13:14:49.554573 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:49.554504    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm" podStartSLOduration=1.5544806740000001 podStartE2EDuration="1.554480674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:14:49.552771392 +0000 UTC m=+2835.659474245" watchObservedRunningTime="2026-11-02 13:14:49.554480674 +0000 UTC m=+2835.661183530"
Nov 02 13:14:49.576418 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:49.576386    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:50.577794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:50.577755    5879 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 02 13:14:50.578240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:50.577810    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm" podUID="c77bd7e8-4b52-427b-90c5-ad7460330a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 02 13:14:50.969576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:50.969541    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:51.576725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:51.576687    5879 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 02 13:14:51.576942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:51.576752    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm" podUID="c77bd7e8-4b52-427b-90c5-ad7460330a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 02 13:14:52.577099 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:52.576981    5879 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 02 13:14:52.577854 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:52.577158    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm" podUID="c77bd7e8-4b52-427b-90c5-ad7460330a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 02 13:14:52.807604 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:52.807560    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:53.575984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:53.575947    5879 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 02 13:14:53.576190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:53.576040    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm" podUID="c77bd7e8-4b52-427b-90c5-ad7460330a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 02 13:14:54.443811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:54.443782    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:54.576151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:54.576114    5879 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 02 13:14:54.576339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:54.576182    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm" podUID="c77bd7e8-4b52-427b-90c5-ad7460330a8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 02 13:14:55.582105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:55.582068    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:55.582554 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:55.582452    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:55.583191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:55.583166    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:14:56.134558 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:56.134496    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:57.861554 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:57.861519    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:59.043637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.043596    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"]
Nov 02 13:14:59.058105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.058072    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31051485-wrbbl"]
Nov 02 13:14:59.655149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.653940    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:59.791293 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.791256    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"]
Nov 02 13:14:59.797031 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.796973    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.803927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.803883    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-dshm\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.804516 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.804059    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-model-cache\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.804516 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.804209    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk9vz\" (UniqueName: \"kubernetes.io/projected/264eaa3c-030e-46be-a090-502091eed5d5-kube-api-access-jk9vz\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.804516 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.804323    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-tmp-dir\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.804516 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.804401    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-home\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.804759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.804589    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/264eaa3c-030e-46be-a090-502091eed5d5-tls-certs\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.814913 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.814882    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9139ec\"/\"openshift-service-ca.crt\""
Nov 02 13:14:59.818632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.818599    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"]
Nov 02 13:14:59.827977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.827942    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9139ec\"/\"kube-root-ca.crt\""
Nov 02 13:14:59.829138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.829105    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9139ec\"/\"default-dockercfg-8vwc4\""
Nov 02 13:14:59.829317 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.829205    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9139ec\"/\"test-model-9139ec-kserve-self-signed-certs\""
Nov 02 13:14:59.851872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.851833    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:14:59.906614 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.906512    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-home\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.906818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.906628    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/264eaa3c-030e-46be-a090-502091eed5d5-tls-certs\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.906818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.906720    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-dshm\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.906942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.906820    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-model-cache\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.906942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.906922    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk9vz\" (UniqueName: \"kubernetes.io/projected/264eaa3c-030e-46be-a090-502091eed5d5-kube-api-access-jk9vz\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.907159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.906985    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-tmp-dir\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.907159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.907062    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-home\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.907342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.907319    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-model-cache\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.907414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.907387    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-tmp-dir\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.909605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.909584    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-dshm\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.910286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.910197    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/264eaa3c-030e-46be-a090-502091eed5d5-tls-certs\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:14:59.940217 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:14:59.936984    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk9vz\" (UniqueName: \"kubernetes.io/projected/264eaa3c-030e-46be-a090-502091eed5d5-kube-api-access-jk9vz\") pod \"test-model-9139ec-kserve-7ff57b865f-mv8k2\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") " pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:15:00.110459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.110139    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4704a4c4-6a62-4a71-90f2-cba9ede3dd5a" path="/var/lib/kubelet/pods/4704a4c4-6a62-4a71-90f2-cba9ede3dd5a/volumes"
Nov 02 13:15:00.130440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.130397    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:15:00.174065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.171818    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"]
Nov 02 13:15:00.179390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.179362    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"
Nov 02 13:15:00.183859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.183830    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"]
Nov 02 13:15:00.190760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.190729    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"
Nov 02 13:15:00.196439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.196123    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"]
Nov 02 13:15:00.204923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.203120    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"]
Nov 02 13:15:00.204923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.204040    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"
Nov 02 13:15:00.204923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.204572    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:15:00.210725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.210699    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"]
Nov 02 13:15:00.212836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.212807    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"
Nov 02 13:15:00.302969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.302937    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"]
Nov 02 13:15:00.304557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.304524    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:00.305185 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.305159    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"
Nov 02 13:15:00.322583 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.322541    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"]
Nov 02 13:15:00.350721 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.327891    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"
Nov 02 13:15:00.354660 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.354624    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8\" (UID: \"c34bf9c9-3f3e-47d2-adaa-8f473dec78be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"
Nov 02 13:15:00.374386 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.354984    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmkqc\" (UniqueName: \"kubernetes.io/projected/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-kube-api-access-tmkqc\") pod \"maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78\" (UID: \"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"
Nov 02 13:15:00.374655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.374634    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsnxl\" (UniqueName: \"kubernetes.io/projected/265f8052-98a4-44a8-95ef-239d37e81c43-kube-api-access-hsnxl\") pod \"maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr\" (UID: \"265f8052-98a4-44a8-95ef-239d37e81c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"
Nov 02 13:15:00.374937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.374917    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvgj6\" (UniqueName: \"kubernetes.io/projected/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-kube-api-access-cvgj6\") pod \"maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8\" (UID: \"c34bf9c9-3f3e-47d2-adaa-8f473dec78be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"
Nov 02 13:15:00.375146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.375125    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c723d82-c700-4def-b8c0-081c5e358b18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl\" (UID: \"4c723d82-c700-4def-b8c0-081c5e358b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"
Nov 02 13:15:00.375328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.375311    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/265f8052-98a4-44a8-95ef-239d37e81c43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr\" (UID: \"265f8052-98a4-44a8-95ef-239d37e81c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"
Nov 02 13:15:00.375563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.375548    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78\" (UID: \"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"
Nov 02 13:15:00.375808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.375789    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dgl7\" (UniqueName: \"kubernetes.io/projected/4c723d82-c700-4def-b8c0-081c5e358b18-kube-api-access-5dgl7\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl\" (UID: \"4c723d82-c700-4def-b8c0-081c5e358b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"
Nov 02 13:15:00.377042 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.376977    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"]
Nov 02 13:15:00.379151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.377277    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"
Nov 02 13:15:00.420116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.420041    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"]
Nov 02 13:15:00.423464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.421524    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"
Nov 02 13:15:00.467200 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.467165    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"]
Nov 02 13:15:00.467376 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.467297    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.476911    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"]
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.476959    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"]
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.476986    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"]
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477038    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"]
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477099    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477123    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"]
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477130    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8\" (UID: \"c34bf9c9-3f3e-47d2-adaa-8f473dec78be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477180    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477177    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/e97847d4-325d-4ce4-874d-199279b16a87-kube-api-access-5th4k\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82\" (UID: \"e97847d4-325d-4ce4-874d-199279b16a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477252    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ccqx\" (UniqueName: \"kubernetes.io/projected/021f486a-e073-4944-8b61-ecece24ff7fe-kube-api-access-5ccqx\") pod \"maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9\" (UID: \"021f486a-e073-4944-8b61-ecece24ff7fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477386    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmkqc\" (UniqueName: \"kubernetes.io/projected/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-kube-api-access-tmkqc\") pod \"maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78\" (UID: \"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477420    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsnxl\" (UniqueName: \"kubernetes.io/projected/265f8052-98a4-44a8-95ef-239d37e81c43-kube-api-access-hsnxl\") pod \"maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr\" (UID: \"265f8052-98a4-44a8-95ef-239d37e81c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477497    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4msn\" (UniqueName: \"kubernetes.io/projected/4863e18b-0942-4e01-94e1-293fd89aa56f-kube-api-access-s4msn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw\" (UID: \"4863e18b-0942-4e01-94e1-293fd89aa56f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477579    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvgj6\" (UniqueName: \"kubernetes.io/projected/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-kube-api-access-cvgj6\") pod \"maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8\" (UID: \"c34bf9c9-3f3e-47d2-adaa-8f473dec78be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477611    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxzn\" (UniqueName: \"kubernetes.io/projected/397a2f80-d67d-43d4-a4f8-35f6e93e111e-kube-api-access-rpxzn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd\" (UID: \"397a2f80-d67d-43d4-a4f8-35f6e93e111e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477650    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c723d82-c700-4def-b8c0-081c5e358b18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl\" (UID: \"4c723d82-c700-4def-b8c0-081c5e358b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477701    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/397a2f80-d67d-43d4-a4f8-35f6e93e111e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd\" (UID: \"397a2f80-d67d-43d4-a4f8-35f6e93e111e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477727    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/265f8052-98a4-44a8-95ef-239d37e81c43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr\" (UID: \"265f8052-98a4-44a8-95ef-239d37e81c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477816    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4863e18b-0942-4e01-94e1-293fd89aa56f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw\" (UID: \"4863e18b-0942-4e01-94e1-293fd89aa56f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477858    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97847d4-325d-4ce4-874d-199279b16a87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82\" (UID: \"e97847d4-325d-4ce4-874d-199279b16a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"
Nov 02 13:15:00.477898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477148    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"]
Nov 02 13:15:00.479388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.477951    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"]
Nov 02 13:15:00.479388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.478811    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c723d82-c700-4def-b8c0-081c5e358b18-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl\" (UID: \"4c723d82-c700-4def-b8c0-081c5e358b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"
Nov 02 13:15:00.479388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.479283    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/265f8052-98a4-44a8-95ef-239d37e81c43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr\" (UID: \"265f8052-98a4-44a8-95ef-239d37e81c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"
Nov 02 13:15:00.479740 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.479722    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8\" (UID: \"c34bf9c9-3f3e-47d2-adaa-8f473dec78be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"
Nov 02 13:15:00.480965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.479947    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/021f486a-e073-4944-8b61-ecece24ff7fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9\" (UID: \"021f486a-e073-4944-8b61-ecece24ff7fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"
Nov 02 13:15:00.480965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.480142    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78\" (UID: \"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"
Nov 02 13:15:00.480965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.480183    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dgl7\" (UniqueName: \"kubernetes.io/projected/4c723d82-c700-4def-b8c0-081c5e358b18-kube-api-access-5dgl7\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl\" (UID: \"4c723d82-c700-4def-b8c0-081c5e358b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"
Nov 02 13:15:00.480965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.480845    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78\" (UID: \"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"
Nov 02 13:15:00.488814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.487433    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"]
Nov 02 13:15:00.488814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.487901    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"]
Nov 02 13:15:00.489350 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.487820    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"
Nov 02 13:15:00.514151 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:15:00.513867    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod264eaa3c_030e_46be_a090_502091eed5d5.slice/crio-17d50e9d6fef932d81a6d658a4740cebfb2c98b16fc844beb9300499b75c080f WatchSource:0}: Error finding container 17d50e9d6fef932d81a6d658a4740cebfb2c98b16fc844beb9300499b75c080f: Status 404 returned error can't find the container with id 17d50e9d6fef932d81a6d658a4740cebfb2c98b16fc844beb9300499b75c080f
Nov 02 13:15:00.514301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.514276    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dgl7\" (UniqueName: \"kubernetes.io/projected/4c723d82-c700-4def-b8c0-081c5e358b18-kube-api-access-5dgl7\") pod \"maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl\" (UID: \"4c723d82-c700-4def-b8c0-081c5e358b18\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"
Nov 02 13:15:00.519406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.519254    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmkqc\" (UniqueName: \"kubernetes.io/projected/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-kube-api-access-tmkqc\") pod \"maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78\" (UID: \"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"
Nov 02 13:15:00.519554 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.519508    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"]
Nov 02 13:15:00.520245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.519792    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"
Nov 02 13:15:00.520890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.520841    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsnxl\" (UniqueName: \"kubernetes.io/projected/265f8052-98a4-44a8-95ef-239d37e81c43-kube-api-access-hsnxl\") pod \"maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr\" (UID: \"265f8052-98a4-44a8-95ef-239d37e81c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"
Nov 02 13:15:00.524718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.524691    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvgj6\" (UniqueName: \"kubernetes.io/projected/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-kube-api-access-cvgj6\") pod \"maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8\" (UID: \"c34bf9c9-3f3e-47d2-adaa-8f473dec78be\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"
Nov 02 13:15:00.526070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.526047    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"]
Nov 02 13:15:00.526166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.526114    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"
Nov 02 13:15:00.532177 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.532153    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"]
Nov 02 13:15:00.532348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.532199    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"]
Nov 02 13:15:00.532348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.532225    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"]
Nov 02 13:15:00.534241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.534218    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"
Nov 02 13:15:00.538622 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.538571    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"]
Nov 02 13:15:00.538800 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.538626    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"]
Nov 02 13:15:00.538800 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.538651    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"]
Nov 02 13:15:00.538800 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.538677    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"
Nov 02 13:15:00.538800 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.538686    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"]
Nov 02 13:15:00.546647 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.546626    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"]
Nov 02 13:15:00.546744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.546729    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"
Nov 02 13:15:00.553655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.553632    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"]
Nov 02 13:15:00.554059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.554041    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"]
Nov 02 13:15:00.554326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.554311    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"]
Nov 02 13:15:00.554517 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.554498    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"]
Nov 02 13:15:00.562174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.559517    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"
Nov 02 13:15:00.565015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.564979    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"]
Nov 02 13:15:00.565227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.565213    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"]
Nov 02 13:15:00.565359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.565348    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"]
Nov 02 13:15:00.565550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.565535    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"
Nov 02 13:15:00.572115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.572042    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"]
Nov 02 13:15:00.582751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.582712    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66cp\" (UniqueName: \"kubernetes.io/projected/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-kube-api-access-x66cp\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt\" (UID: \"5ffec64c-ff1f-42df-83a2-3d45751f8bb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"
Nov 02 13:15:00.582933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.582795    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktq7m\" (UniqueName: \"kubernetes.io/projected/d8964b32-2de3-4bb6-b25c-7d7a561d0251-kube-api-access-ktq7m\") pod \"maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c\" (UID: \"d8964b32-2de3-4bb6-b25c-7d7a561d0251\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"
Nov 02 13:15:00.582933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.582834    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcl8h\" (UniqueName: \"kubernetes.io/projected/20982710-74e3-4acb-af83-0f1d76338674-kube-api-access-hcl8h\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg\" (UID: \"20982710-74e3-4acb-af83-0f1d76338674\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"
Nov 02 13:15:00.582933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.582898    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4msn\" (UniqueName: \"kubernetes.io/projected/4863e18b-0942-4e01-94e1-293fd89aa56f-kube-api-access-s4msn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw\" (UID: \"4863e18b-0942-4e01-94e1-293fd89aa56f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"
Nov 02 13:15:00.583144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.582937    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2nc5\" (UniqueName: \"kubernetes.io/projected/031e33ac-9ee4-4153-9bdb-5289b569a146-kube-api-access-x2nc5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2\" (UID: \"031e33ac-9ee4-4153-9bdb-5289b569a146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"
Nov 02 13:15:00.583144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.582986    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt\" (UID: \"9025c2c7-57b9-439c-bbf8-91a5fd1433c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"
Nov 02 13:15:00.583144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583088    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxzn\" (UniqueName: \"kubernetes.io/projected/397a2f80-d67d-43d4-a4f8-35f6e93e111e-kube-api-access-rpxzn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd\" (UID: \"397a2f80-d67d-43d4-a4f8-35f6e93e111e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"
Nov 02 13:15:00.583314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583145    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23754db5-04c9-48a5-841a-433830943be2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5\" (UID: \"23754db5-04c9-48a5-841a-433830943be2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"
Nov 02 13:15:00.583314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583187    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk\" (UID: \"9262bfd3-6bbc-4161-afa6-93fd1a6639a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"
Nov 02 13:15:00.583314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583216    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/397a2f80-d67d-43d4-a4f8-35f6e93e111e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd\" (UID: \"397a2f80-d67d-43d4-a4f8-35f6e93e111e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"
Nov 02 13:15:00.583314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583246    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfv62\" (UniqueName: \"kubernetes.io/projected/e9110a44-4129-4c65-9de1-f2574890c394-kube-api-access-nfv62\") pod \"maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv\" (UID: \"e9110a44-4129-4c65-9de1-f2574890c394\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"
Nov 02 13:15:00.583314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583275    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8964b32-2de3-4bb6-b25c-7d7a561d0251-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c\" (UID: \"d8964b32-2de3-4bb6-b25c-7d7a561d0251\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"
Nov 02 13:15:00.583580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583319    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwdf\" (UniqueName: \"kubernetes.io/projected/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-kube-api-access-bfwdf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt\" (UID: \"9025c2c7-57b9-439c-bbf8-91a5fd1433c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"
Nov 02 13:15:00.583580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583360    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4863e18b-0942-4e01-94e1-293fd89aa56f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw\" (UID: \"4863e18b-0942-4e01-94e1-293fd89aa56f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"
Nov 02 13:15:00.583580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583404    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw72z\" (UniqueName: \"kubernetes.io/projected/23754db5-04c9-48a5-841a-433830943be2-kube-api-access-hw72z\") pod \"maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5\" (UID: \"23754db5-04c9-48a5-841a-433830943be2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"
Nov 02 13:15:00.583580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583430    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97847d4-325d-4ce4-874d-199279b16a87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82\" (UID: \"e97847d4-325d-4ce4-874d-199279b16a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"
Nov 02 13:15:00.583580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583477    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9110a44-4129-4c65-9de1-f2574890c394-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv\" (UID: \"e9110a44-4129-4c65-9de1-f2574890c394\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"
Nov 02 13:15:00.583580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583501    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/021f486a-e073-4944-8b61-ecece24ff7fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9\" (UID: \"021f486a-e073-4944-8b61-ecece24ff7fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"
Nov 02 13:15:00.583580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583539    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m7zb\" (UniqueName: \"kubernetes.io/projected/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-kube-api-access-8m7zb\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk\" (UID: \"9262bfd3-6bbc-4161-afa6-93fd1a6639a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"
Nov 02 13:15:00.583927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583619    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba4d701-2d78-40cf-bacc-8206cc16e923-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz\" (UID: \"2ba4d701-2d78-40cf-bacc-8206cc16e923\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"
Nov 02 13:15:00.583927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583657    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/e97847d4-325d-4ce4-874d-199279b16a87-kube-api-access-5th4k\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82\" (UID: \"e97847d4-325d-4ce4-874d-199279b16a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"
Nov 02 13:15:00.583927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583705    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ccqx\" (UniqueName: \"kubernetes.io/projected/021f486a-e073-4944-8b61-ecece24ff7fe-kube-api-access-5ccqx\") pod \"maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9\" (UID: \"021f486a-e073-4944-8b61-ecece24ff7fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"
Nov 02 13:15:00.583927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583758    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt\" (UID: \"5ffec64c-ff1f-42df-83a2-3d45751f8bb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"
Nov 02 13:15:00.583927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583790    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7dt7\" (UniqueName: \"kubernetes.io/projected/2ba4d701-2d78-40cf-bacc-8206cc16e923-kube-api-access-m7dt7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz\" (UID: \"2ba4d701-2d78-40cf-bacc-8206cc16e923\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"
Nov 02 13:15:00.583927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583819    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4tc8\" (UniqueName: \"kubernetes.io/projected/e518dcfa-edd5-4c9c-b128-3504862ab477-kube-api-access-b4tc8\") pod \"maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq\" (UID: \"e518dcfa-edd5-4c9c-b128-3504862ab477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"
Nov 02 13:15:00.583927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583848    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031e33ac-9ee4-4153-9bdb-5289b569a146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2\" (UID: \"031e33ac-9ee4-4153-9bdb-5289b569a146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"
Nov 02 13:15:00.583927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583924    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e518dcfa-edd5-4c9c-b128-3504862ab477-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq\" (UID: \"e518dcfa-edd5-4c9c-b128-3504862ab477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"
Nov 02 13:15:00.584396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.583952    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20982710-74e3-4acb-af83-0f1d76338674-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg\" (UID: \"20982710-74e3-4acb-af83-0f1d76338674\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"
Nov 02 13:15:00.586621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.585264    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/397a2f80-d67d-43d4-a4f8-35f6e93e111e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd\" (UID: \"397a2f80-d67d-43d4-a4f8-35f6e93e111e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"
Nov 02 13:15:00.586621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.585931    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4863e18b-0942-4e01-94e1-293fd89aa56f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw\" (UID: \"4863e18b-0942-4e01-94e1-293fd89aa56f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"
Nov 02 13:15:00.586621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.586575    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97847d4-325d-4ce4-874d-199279b16a87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82\" (UID: \"e97847d4-325d-4ce4-874d-199279b16a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"
Nov 02 13:15:00.587097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.587077    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/021f486a-e073-4944-8b61-ecece24ff7fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9\" (UID: \"021f486a-e073-4944-8b61-ecece24ff7fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"
Nov 02 13:15:00.599751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.599720    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2" event={"ID":"264eaa3c-030e-46be-a090-502091eed5d5","Type":"ContainerStarted","Data":"17d50e9d6fef932d81a6d658a4740cebfb2c98b16fc844beb9300499b75c080f"}
Nov 02 13:15:00.610367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.610331    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4msn\" (UniqueName: \"kubernetes.io/projected/4863e18b-0942-4e01-94e1-293fd89aa56f-kube-api-access-s4msn\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw\" (UID: \"4863e18b-0942-4e01-94e1-293fd89aa56f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"
Nov 02 13:15:00.611126 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.611101    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxzn\" (UniqueName: \"kubernetes.io/projected/397a2f80-d67d-43d4-a4f8-35f6e93e111e-kube-api-access-rpxzn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd\" (UID: \"397a2f80-d67d-43d4-a4f8-35f6e93e111e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"
Nov 02 13:15:00.614370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.614349    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ccqx\" (UniqueName: \"kubernetes.io/projected/021f486a-e073-4944-8b61-ecece24ff7fe-kube-api-access-5ccqx\") pod \"maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9\" (UID: \"021f486a-e073-4944-8b61-ecece24ff7fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"
Nov 02 13:15:00.619406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.619383    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/e97847d4-325d-4ce4-874d-199279b16a87-kube-api-access-5th4k\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82\" (UID: \"e97847d4-325d-4ce4-874d-199279b16a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"
Nov 02 13:15:00.653787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.653754    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691231    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2nc5\" (UniqueName: \"kubernetes.io/projected/031e33ac-9ee4-4153-9bdb-5289b569a146-kube-api-access-x2nc5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2\" (UID: \"031e33ac-9ee4-4153-9bdb-5289b569a146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691299    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt\" (UID: \"9025c2c7-57b9-439c-bbf8-91a5fd1433c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691437    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23754db5-04c9-48a5-841a-433830943be2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5\" (UID: \"23754db5-04c9-48a5-841a-433830943be2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691474    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk\" (UID: \"9262bfd3-6bbc-4161-afa6-93fd1a6639a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691508    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfv62\" (UniqueName: \"kubernetes.io/projected/e9110a44-4129-4c65-9de1-f2574890c394-kube-api-access-nfv62\") pod \"maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv\" (UID: \"e9110a44-4129-4c65-9de1-f2574890c394\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691534    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8964b32-2de3-4bb6-b25c-7d7a561d0251-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c\" (UID: \"d8964b32-2de3-4bb6-b25c-7d7a561d0251\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691592    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwdf\" (UniqueName: \"kubernetes.io/projected/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-kube-api-access-bfwdf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt\" (UID: \"9025c2c7-57b9-439c-bbf8-91a5fd1433c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691649    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw72z\" (UniqueName: \"kubernetes.io/projected/23754db5-04c9-48a5-841a-433830943be2-kube-api-access-hw72z\") pod \"maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5\" (UID: \"23754db5-04c9-48a5-841a-433830943be2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691702    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9110a44-4129-4c65-9de1-f2574890c394-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv\" (UID: \"e9110a44-4129-4c65-9de1-f2574890c394\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691728    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m7zb\" (UniqueName: \"kubernetes.io/projected/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-kube-api-access-8m7zb\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk\" (UID: \"9262bfd3-6bbc-4161-afa6-93fd1a6639a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691789    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba4d701-2d78-40cf-bacc-8206cc16e923-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz\" (UID: \"2ba4d701-2d78-40cf-bacc-8206cc16e923\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691858    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt\" (UID: \"5ffec64c-ff1f-42df-83a2-3d45751f8bb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691890    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7dt7\" (UniqueName: \"kubernetes.io/projected/2ba4d701-2d78-40cf-bacc-8206cc16e923-kube-api-access-m7dt7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz\" (UID: \"2ba4d701-2d78-40cf-bacc-8206cc16e923\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691918    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4tc8\" (UniqueName: \"kubernetes.io/projected/e518dcfa-edd5-4c9c-b128-3504862ab477-kube-api-access-b4tc8\") pod \"maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq\" (UID: \"e518dcfa-edd5-4c9c-b128-3504862ab477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.691947    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031e33ac-9ee4-4153-9bdb-5289b569a146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2\" (UID: \"031e33ac-9ee4-4153-9bdb-5289b569a146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.692049    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e518dcfa-edd5-4c9c-b128-3504862ab477-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq\" (UID: \"e518dcfa-edd5-4c9c-b128-3504862ab477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.692081    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20982710-74e3-4acb-af83-0f1d76338674-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg\" (UID: \"20982710-74e3-4acb-af83-0f1d76338674\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.692123    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66cp\" (UniqueName: \"kubernetes.io/projected/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-kube-api-access-x66cp\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt\" (UID: \"5ffec64c-ff1f-42df-83a2-3d45751f8bb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.692193    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktq7m\" (UniqueName: \"kubernetes.io/projected/d8964b32-2de3-4bb6-b25c-7d7a561d0251-kube-api-access-ktq7m\") pod \"maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c\" (UID: \"d8964b32-2de3-4bb6-b25c-7d7a561d0251\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.692228    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcl8h\" (UniqueName: \"kubernetes.io/projected/20982710-74e3-4acb-af83-0f1d76338674-kube-api-access-hcl8h\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg\" (UID: \"20982710-74e3-4acb-af83-0f1d76338674\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.692449    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk\" (UID: \"9262bfd3-6bbc-4161-afa6-93fd1a6639a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.693320    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt\" (UID: \"9025c2c7-57b9-439c-bbf8-91a5fd1433c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"
Nov 02 13:15:00.693696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.693373    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8964b32-2de3-4bb6-b25c-7d7a561d0251-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c\" (UID: \"d8964b32-2de3-4bb6-b25c-7d7a561d0251\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"
Nov 02 13:15:00.695216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.694553    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9110a44-4129-4c65-9de1-f2574890c394-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv\" (UID: \"e9110a44-4129-4c65-9de1-f2574890c394\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"
Nov 02 13:15:00.697089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.695516    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba4d701-2d78-40cf-bacc-8206cc16e923-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz\" (UID: \"2ba4d701-2d78-40cf-bacc-8206cc16e923\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"
Nov 02 13:15:00.697089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.695569    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031e33ac-9ee4-4153-9bdb-5289b569a146-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2\" (UID: \"031e33ac-9ee4-4153-9bdb-5289b569a146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"
Nov 02 13:15:00.697089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.696282    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt\" (UID: \"5ffec64c-ff1f-42df-83a2-3d45751f8bb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"
Nov 02 13:15:00.697089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.696284    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e518dcfa-edd5-4c9c-b128-3504862ab477-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq\" (UID: \"e518dcfa-edd5-4c9c-b128-3504862ab477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"
Nov 02 13:15:00.698191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.697647    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23754db5-04c9-48a5-841a-433830943be2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5\" (UID: \"23754db5-04c9-48a5-841a-433830943be2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"
Nov 02 13:15:00.698801 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.698779    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20982710-74e3-4acb-af83-0f1d76338674-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg\" (UID: \"20982710-74e3-4acb-af83-0f1d76338674\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"
Nov 02 13:15:00.710922 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.710649    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"
Nov 02 13:15:00.711630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.711591    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:00.722879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.721789    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcl8h\" (UniqueName: \"kubernetes.io/projected/20982710-74e3-4acb-af83-0f1d76338674-kube-api-access-hcl8h\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg\" (UID: \"20982710-74e3-4acb-af83-0f1d76338674\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"
Nov 02 13:15:00.732861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.726783    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"
Nov 02 13:15:00.734641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.734122    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4tc8\" (UniqueName: \"kubernetes.io/projected/e518dcfa-edd5-4c9c-b128-3504862ab477-kube-api-access-b4tc8\") pod \"maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq\" (UID: \"e518dcfa-edd5-4c9c-b128-3504862ab477\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"
Nov 02 13:15:00.736632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.736531    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"
Nov 02 13:15:00.738287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.737560    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2nc5\" (UniqueName: \"kubernetes.io/projected/031e33ac-9ee4-4153-9bdb-5289b569a146-kube-api-access-x2nc5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2\" (UID: \"031e33ac-9ee4-4153-9bdb-5289b569a146\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"
Nov 02 13:15:00.738287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.738227    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktq7m\" (UniqueName: \"kubernetes.io/projected/d8964b32-2de3-4bb6-b25c-7d7a561d0251-kube-api-access-ktq7m\") pod \"maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c\" (UID: \"d8964b32-2de3-4bb6-b25c-7d7a561d0251\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"
Nov 02 13:15:00.739331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.739260    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66cp\" (UniqueName: \"kubernetes.io/projected/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-kube-api-access-x66cp\") pod \"maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt\" (UID: \"5ffec64c-ff1f-42df-83a2-3d45751f8bb7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"
Nov 02 13:15:00.740083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.739968    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m7zb\" (UniqueName: \"kubernetes.io/projected/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-kube-api-access-8m7zb\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk\" (UID: \"9262bfd3-6bbc-4161-afa6-93fd1a6639a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"
Nov 02 13:15:00.741485 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.741450    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw72z\" (UniqueName: \"kubernetes.io/projected/23754db5-04c9-48a5-841a-433830943be2-kube-api-access-hw72z\") pod \"maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5\" (UID: \"23754db5-04c9-48a5-841a-433830943be2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"
Nov 02 13:15:00.741594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.741536    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7dt7\" (UniqueName: \"kubernetes.io/projected/2ba4d701-2d78-40cf-bacc-8206cc16e923-kube-api-access-m7dt7\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz\" (UID: \"2ba4d701-2d78-40cf-bacc-8206cc16e923\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"
Nov 02 13:15:00.743920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.742905    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwdf\" (UniqueName: \"kubernetes.io/projected/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-kube-api-access-bfwdf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt\" (UID: \"9025c2c7-57b9-439c-bbf8-91a5fd1433c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"
Nov 02 13:15:00.743920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.744408    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfv62\" (UniqueName: \"kubernetes.io/projected/e9110a44-4129-4c65-9de1-f2574890c394-kube-api-access-nfv62\") pod \"maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv\" (UID: \"e9110a44-4129-4c65-9de1-f2574890c394\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"
Nov 02 13:15:00.747129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.746085    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"
Nov 02 13:15:00.747129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.746903    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"
Nov 02 13:15:00.770432 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.758060    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"
Nov 02 13:15:00.777199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.777166    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"
Nov 02 13:15:00.815821 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.797439    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"
Nov 02 13:15:00.826519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.826484    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"
Nov 02 13:15:00.828284 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.828261    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"
Nov 02 13:15:00.840723 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.839019    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"
Nov 02 13:15:00.851599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.851027    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"
Nov 02 13:15:00.863358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.863164    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"
Nov 02 13:15:00.950397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.950215    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"
Nov 02 13:15:00.968350 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.968276    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"
Nov 02 13:15:00.982367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:00.980755    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"
Nov 02 13:15:01.029263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:01.027538    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"
Nov 02 13:15:01.123428 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:01.123200    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"]
Nov 02 13:15:01.182295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:01.181931    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:01.569531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:01.569481    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"]
Nov 02 13:15:01.650737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:01.650115    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2" event={"ID":"264eaa3c-030e-46be-a090-502091eed5d5","Type":"ContainerStarted","Data":"1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5"}
Nov 02 13:15:01.662573 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:01.662510    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr" event={"ID":"265f8052-98a4-44a8-95ef-239d37e81c43","Type":"ContainerStarted","Data":"892ec15c296f58c0de372c93dcc0862f0a92b98fd2b7fe941da76586e2bcd3d1"}
Nov 02 13:15:01.665163 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:01.665057    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78" event={"ID":"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05","Type":"ContainerStarted","Data":"03baab5ade87c4e823c19e42848001904cc644726519b04576a6a2a7ad720f39"}
Nov 02 13:15:01.746065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:01.743557    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:01.924056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:01.923833    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:01.978948 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:01.930673    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"]
Nov 02 13:15:01.978948 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:01.964676    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2" podStartSLOduration=2.9646530049999997 podStartE2EDuration="2.964653005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:01.725748627 +0000 UTC m=+2847.832451480" watchObservedRunningTime="2026-11-02 13:15:01.964653005 +0000 UTC m=+2848.071355861"
Nov 02 13:15:02.002742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.001258    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"]
Nov 02 13:15:02.094812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.094760    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"]
Nov 02 13:15:02.131431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.131382    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:15:02.378982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.378892    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"]
Nov 02 13:15:02.390915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.388438    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"]
Nov 02 13:15:02.405593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.405556    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"]
Nov 02 13:15:02.411739 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:15:02.411598    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4863e18b_0942_4e01_94e1_293fd89aa56f.slice/crio-141b33d841cf3b6c4b1fd1eb35f2a16d701a36849442005c0cba7b3626f25a4b WatchSource:0}: Error finding container 141b33d841cf3b6c4b1fd1eb35f2a16d701a36849442005c0cba7b3626f25a4b: Status 404 returned error can't find the container with id 141b33d841cf3b6c4b1fd1eb35f2a16d701a36849442005c0cba7b3626f25a4b
Nov 02 13:15:02.511109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.511064    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"]
Nov 02 13:15:02.599408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.599316    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"]
Nov 02 13:15:02.609327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.609224    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"]
Nov 02 13:15:02.611624 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:15:02.611456    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9262bfd3_6bbc_4161_afa6_93fd1a6639a1.slice/crio-7eb3299a0708771337374e08d2f77cb5ffa394a418b34087fb81901287ca58a5 WatchSource:0}: Error finding container 7eb3299a0708771337374e08d2f77cb5ffa394a418b34087fb81901287ca58a5: Status 404 returned error can't find the container with id 7eb3299a0708771337374e08d2f77cb5ffa394a418b34087fb81901287ca58a5
Nov 02 13:15:02.612209 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:15:02.612182    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode518dcfa_edd5_4c9c_b128_3504862ab477.slice/crio-b44e823ba3f432cdd02f24f294af723f895a57b1491c45cad8755490aec12612 WatchSource:0}: Error finding container b44e823ba3f432cdd02f24f294af723f895a57b1491c45cad8755490aec12612: Status 404 returned error can't find the container with id b44e823ba3f432cdd02f24f294af723f895a57b1491c45cad8755490aec12612
Nov 02 13:15:02.699144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.699062    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz" event={"ID":"2ba4d701-2d78-40cf-bacc-8206cc16e923","Type":"ContainerStarted","Data":"b9c5993aa6de4846a104f83ae08a1e4e955eb4fed655aa3685e6d78fbed4ef4f"}
Nov 02 13:15:02.700827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.700798    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg" event={"ID":"20982710-74e3-4acb-af83-0f1d76338674","Type":"ContainerStarted","Data":"5cc8035017793d5ddceff123f708f31c4c0816d4f063a546434923439fd77c3a"}
Nov 02 13:15:02.704123 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.703332    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78" event={"ID":"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05","Type":"ContainerStarted","Data":"b9ff7295da6964d240ec3d6fa72462f503584eddea6e494af5f91fd37d399884"}
Nov 02 13:15:02.706308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.706282    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9" event={"ID":"021f486a-e073-4944-8b61-ecece24ff7fe","Type":"ContainerStarted","Data":"dac3eb7e9866f45d8a586c3296ab1dc2db00c5f768492a4b7defc76632fed732"}
Nov 02 13:15:02.707932 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.707906    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8" event={"ID":"c34bf9c9-3f3e-47d2-adaa-8f473dec78be","Type":"ContainerStarted","Data":"4883e762c7b309ab20ed049b5a7a1699444938adc49f9716b725606ff26c02bb"}
Nov 02 13:15:02.709146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.709118    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl" event={"ID":"4c723d82-c700-4def-b8c0-081c5e358b18","Type":"ContainerStarted","Data":"fad7a6acebad6aa8811f2d71c71e4e2ece2e76fb5d3ae438a6c1f64bf4a108ae"}
Nov 02 13:15:02.711295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.711240    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw" event={"ID":"4863e18b-0942-4e01-94e1-293fd89aa56f","Type":"ContainerStarted","Data":"141b33d841cf3b6c4b1fd1eb35f2a16d701a36849442005c0cba7b3626f25a4b"}
Nov 02 13:15:02.713147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.713124    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk" event={"ID":"9262bfd3-6bbc-4161-afa6-93fd1a6639a1","Type":"ContainerStarted","Data":"7eb3299a0708771337374e08d2f77cb5ffa394a418b34087fb81901287ca58a5"}
Nov 02 13:15:02.714382 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.714358    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt" event={"ID":"5ffec64c-ff1f-42df-83a2-3d45751f8bb7","Type":"ContainerStarted","Data":"138e558331c36c3d42d593a7b29d02ab7320b740fba685255ffd6d7c439742c7"}
Nov 02 13:15:02.716042 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.715980    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq" event={"ID":"e518dcfa-edd5-4c9c-b128-3504862ab477","Type":"ContainerStarted","Data":"b44e823ba3f432cdd02f24f294af723f895a57b1491c45cad8755490aec12612"}
Nov 02 13:15:02.754324 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.754263    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78" podStartSLOduration=2.754244613 podStartE2EDuration="2.754244613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:02.749615612 +0000 UTC m=+2848.856318464" watchObservedRunningTime="2026-11-02 13:15:02.754244613 +0000 UTC m=+2848.860947464"
Nov 02 13:15:02.768158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.768089    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"]
Nov 02 13:15:02.787510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.787461    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"]
Nov 02 13:15:02.895345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.895317    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"]
Nov 02 13:15:02.904340 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:15:02.904302    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397a2f80_d67d_43d4_a4f8_35f6e93e111e.slice/crio-6e386d0ae35b3f86ba53416a864471f27b2763314351c12b613980d8386456f9 WatchSource:0}: Error finding container 6e386d0ae35b3f86ba53416a864471f27b2763314351c12b613980d8386456f9: Status 404 returned error can't find the container with id 6e386d0ae35b3f86ba53416a864471f27b2763314351c12b613980d8386456f9
Nov 02 13:15:02.907772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.907744    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"]
Nov 02 13:15:02.920212 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.920133    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"]
Nov 02 13:15:02.926600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.926569    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"]
Nov 02 13:15:02.930727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:02.930702    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"]
Nov 02 13:15:02.988114 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:15:02.988072    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9025c2c7_57b9_439c_bbf8_91a5fd1433c0.slice/crio-6ddb8498ebb26b073094d5cafb0348df67be139f0b1be1e26ed5ff901573556f WatchSource:0}: Error finding container 6ddb8498ebb26b073094d5cafb0348df67be139f0b1be1e26ed5ff901573556f: Status 404 returned error can't find the container with id 6ddb8498ebb26b073094d5cafb0348df67be139f0b1be1e26ed5ff901573556f
Nov 02 13:15:02.995683 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:15:02.995657    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97847d4_325d_4ce4_874d_199279b16a87.slice/crio-7eb781d193fd948a5a983c132aa545d836f84ec06eea29c213039a277acd8def WatchSource:0}: Error finding container 7eb781d193fd948a5a983c132aa545d836f84ec06eea29c213039a277acd8def: Status 404 returned error can't find the container with id 7eb781d193fd948a5a983c132aa545d836f84ec06eea29c213039a277acd8def
Nov 02 13:15:03.002764 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:15:03.002732    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9110a44_4129_4c65_9de1_f2574890c394.slice/crio-fb7cf0e6b4716e7b0d6e33941f724a9cf0b0bacd95d1654941cab61680e641c7 WatchSource:0}: Error finding container fb7cf0e6b4716e7b0d6e33941f724a9cf0b0bacd95d1654941cab61680e641c7: Status 404 returned error can't find the container with id fb7cf0e6b4716e7b0d6e33941f724a9cf0b0bacd95d1654941cab61680e641c7
Nov 02 13:15:03.101697 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:15:03.101650    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8964b32_2de3_4bb6_b25c_7d7a561d0251.slice/crio-dab19a4095d363b0727b081d93d4d44a2a49c4248a5c72b1306636efeb7fb6a6 WatchSource:0}: Error finding container dab19a4095d363b0727b081d93d4d44a2a49c4248a5c72b1306636efeb7fb6a6: Status 404 returned error can't find the container with id dab19a4095d363b0727b081d93d4d44a2a49c4248a5c72b1306636efeb7fb6a6
Nov 02 13:15:03.665703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.665633    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:03.726584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.726532    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz" event={"ID":"2ba4d701-2d78-40cf-bacc-8206cc16e923","Type":"ContainerStarted","Data":"359f829e2449079a7110f52e2a29e65401614c770e68ebe7e796f749cbe0edcb"}
Nov 02 13:15:03.728799 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.728769    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg" event={"ID":"20982710-74e3-4acb-af83-0f1d76338674","Type":"ContainerStarted","Data":"ee700de62756d55f9630a10bd780fcbb46a4d25dca96af269c951513b392de6f"}
Nov 02 13:15:03.730832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.730797    5879 generic.go:356] "Generic (PLEG): container finished" podID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" containerID="b9ff7295da6964d240ec3d6fa72462f503584eddea6e494af5f91fd37d399884" exitCode=60
Nov 02 13:15:03.731089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.730877    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78" event={"ID":"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05","Type":"ContainerDied","Data":"b9ff7295da6964d240ec3d6fa72462f503584eddea6e494af5f91fd37d399884"}
Nov 02 13:15:03.731234 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.731212    5879 scope.go:122] "RemoveContainer" containerID="b9ff7295da6964d240ec3d6fa72462f503584eddea6e494af5f91fd37d399884"
Nov 02 13:15:03.732855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.732829    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt" event={"ID":"9025c2c7-57b9-439c-bbf8-91a5fd1433c0","Type":"ContainerStarted","Data":"6ddb8498ebb26b073094d5cafb0348df67be139f0b1be1e26ed5ff901573556f"}
Nov 02 13:15:03.734571 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.734543    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv" event={"ID":"e9110a44-4129-4c65-9de1-f2574890c394","Type":"ContainerStarted","Data":"fb7cf0e6b4716e7b0d6e33941f724a9cf0b0bacd95d1654941cab61680e641c7"}
Nov 02 13:15:03.735944 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.735919    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2" event={"ID":"031e33ac-9ee4-4153-9bdb-5289b569a146","Type":"ContainerStarted","Data":"ec80af19e32f2b893643f2a1065f9276a6174bd093ffb46a06e39e9debff7b37"}
Nov 02 13:15:03.737889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.737834    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9" event={"ID":"021f486a-e073-4944-8b61-ecece24ff7fe","Type":"ContainerStarted","Data":"e45aabf958ccba68367978af9d41a7333b635dd72c00c613b8bce81560b5704c"}
Nov 02 13:15:03.739877 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.739803    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd" event={"ID":"397a2f80-d67d-43d4-a4f8-35f6e93e111e","Type":"ContainerStarted","Data":"6e386d0ae35b3f86ba53416a864471f27b2763314351c12b613980d8386456f9"}
Nov 02 13:15:03.748348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.741702    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr" event={"ID":"265f8052-98a4-44a8-95ef-239d37e81c43","Type":"ContainerStarted","Data":"e8eb5169a7c3f7fac80842c3d82d133e1760cbee5437502ddb40e5fd2da3ea82"}
Nov 02 13:15:03.748348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.747628    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8" event={"ID":"c34bf9c9-3f3e-47d2-adaa-8f473dec78be","Type":"ContainerStarted","Data":"75ec9c912b3d971cf143cdb649e0901ee0a227873551197ca97029634f7442de"}
Nov 02 13:15:03.751226 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.751170    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5" event={"ID":"23754db5-04c9-48a5-841a-433830943be2","Type":"ContainerStarted","Data":"c7480895742266210a2bcbfed68be71eff661cb2f41f16793026abeabf1fad4e"}
Nov 02 13:15:03.756616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.756509    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl" event={"ID":"4c723d82-c700-4def-b8c0-081c5e358b18","Type":"ContainerStarted","Data":"e91cb4eb4d0de76fdd5bb737d0202215e394a1a990e2a38f01ce89fbefb57286"}
Nov 02 13:15:03.761810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.761760    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82" event={"ID":"e97847d4-325d-4ce4-874d-199279b16a87","Type":"ContainerStarted","Data":"7eb781d193fd948a5a983c132aa545d836f84ec06eea29c213039a277acd8def"}
Nov 02 13:15:03.765045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.764814    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw" event={"ID":"4863e18b-0942-4e01-94e1-293fd89aa56f","Type":"ContainerStarted","Data":"9366f1256db9bced1a747174c1fba36230480843e5f5796b5de294293d0d7158"}
Nov 02 13:15:03.767795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.767764    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk" event={"ID":"9262bfd3-6bbc-4161-afa6-93fd1a6639a1","Type":"ContainerStarted","Data":"8ac42d71b010e80747a49587decbaf98fe535b6570a75cb73abc1aeb866fb106"}
Nov 02 13:15:03.770134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.770095    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt" event={"ID":"5ffec64c-ff1f-42df-83a2-3d45751f8bb7","Type":"ContainerStarted","Data":"dace30a2c4be706fd06c9340bdc1148da2ba0c3c44cdf759e084806b16347536"}
Nov 02 13:15:03.772558 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.772531    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq" event={"ID":"e518dcfa-edd5-4c9c-b128-3504862ab477","Type":"ContainerStarted","Data":"88d3e058cafe80aa5d3068cc356467e335c12566e606752ad7ec904fdc9fef87"}
Nov 02 13:15:03.774868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.774837    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" event={"ID":"d8964b32-2de3-4bb6-b25c-7d7a561d0251","Type":"ContainerStarted","Data":"dab19a4095d363b0727b081d93d4d44a2a49c4248a5c72b1306636efeb7fb6a6"}
Nov 02 13:15:03.781942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.781894    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz" podStartSLOduration=3.781879596 podStartE2EDuration="3.781879596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:03.775710387 +0000 UTC m=+2849.882413238" watchObservedRunningTime="2026-11-02 13:15:03.781879596 +0000 UTC m=+2849.888582446"
Nov 02 13:15:03.828621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.828561    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8" podStartSLOduration=3.82854753 podStartE2EDuration="3.82854753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:03.825686339 +0000 UTC m=+2849.932389193" watchObservedRunningTime="2026-11-02 13:15:03.82854753 +0000 UTC m=+2849.935250380"
Nov 02 13:15:03.878852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.878802    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq" podStartSLOduration=3.878787559 podStartE2EDuration="3.878787559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:03.876285148 +0000 UTC m=+2849.982987999" watchObservedRunningTime="2026-11-02 13:15:03.878787559 +0000 UTC m=+2849.985490409"
Nov 02 13:15:03.926962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.926903    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk" podStartSLOduration=3.926883845 podStartE2EDuration="3.926883845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:03.925171952 +0000 UTC m=+2850.031874804" watchObservedRunningTime="2026-11-02 13:15:03.926883845 +0000 UTC m=+2850.033586699"
Nov 02 13:15:03.975504 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:03.975441    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl" podStartSLOduration=3.975425764 podStartE2EDuration="3.975425764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:03.971937177 +0000 UTC m=+2850.078640030" watchObservedRunningTime="2026-11-02 13:15:03.975425764 +0000 UTC m=+2850.082128646"
Nov 02 13:15:04.032369 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.032311    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw" podStartSLOduration=4.032295126 podStartE2EDuration="4.032295126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.03019041 +0000 UTC m=+2850.136893263" watchObservedRunningTime="2026-11-02 13:15:04.032295126 +0000 UTC m=+2850.138998022"
Nov 02 13:15:04.089937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.089636    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr" podStartSLOduration=4.089609227 podStartE2EDuration="4.089609227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.082657475 +0000 UTC m=+2850.189360327" watchObservedRunningTime="2026-11-02 13:15:04.089609227 +0000 UTC m=+2850.196312080"
Nov 02 13:15:04.143003 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.142935    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt" podStartSLOduration=4.142918846 podStartE2EDuration="4.142918846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.13807615 +0000 UTC m=+2850.244779000" watchObservedRunningTime="2026-11-02 13:15:04.142918846 +0000 UTC m=+2850.249621700"
Nov 02 13:15:04.196212 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.196149    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9" podStartSLOduration=4.196127042 podStartE2EDuration="4.196127042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.189105028 +0000 UTC m=+2850.295807872" watchObservedRunningTime="2026-11-02 13:15:04.196127042 +0000 UTC m=+2850.302829895"
Nov 02 13:15:04.292234 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.292171    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg" podStartSLOduration=4.292155024 podStartE2EDuration="4.292155024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.287355185 +0000 UTC m=+2850.394058036" watchObservedRunningTime="2026-11-02 13:15:04.292155024 +0000 UTC m=+2850.398857888"
Nov 02 13:15:04.791867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.791823    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82" event={"ID":"e97847d4-325d-4ce4-874d-199279b16a87","Type":"ContainerStarted","Data":"7b93b154c5b0c02c714ed5901151c4e5d5b17e416e9ea9d82bb726d25168bc68"}
Nov 02 13:15:04.793725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.793695    5879 generic.go:356] "Generic (PLEG): container finished" podID="4863e18b-0942-4e01-94e1-293fd89aa56f" containerID="9366f1256db9bced1a747174c1fba36230480843e5f5796b5de294293d0d7158" exitCode=60
Nov 02 13:15:04.793866 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.793771    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw" event={"ID":"4863e18b-0942-4e01-94e1-293fd89aa56f","Type":"ContainerDied","Data":"9366f1256db9bced1a747174c1fba36230480843e5f5796b5de294293d0d7158"}
Nov 02 13:15:04.794033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.794017    5879 scope.go:122] "RemoveContainer" containerID="9366f1256db9bced1a747174c1fba36230480843e5f5796b5de294293d0d7158"
Nov 02 13:15:04.795742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.795721    5879 generic.go:356] "Generic (PLEG): container finished" podID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" containerID="8ac42d71b010e80747a49587decbaf98fe535b6570a75cb73abc1aeb866fb106" exitCode=60
Nov 02 13:15:04.795815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.795795    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk" event={"ID":"9262bfd3-6bbc-4161-afa6-93fd1a6639a1","Type":"ContainerDied","Data":"8ac42d71b010e80747a49587decbaf98fe535b6570a75cb73abc1aeb866fb106"}
Nov 02 13:15:04.796070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.796055    5879 scope.go:122] "RemoveContainer" containerID="8ac42d71b010e80747a49587decbaf98fe535b6570a75cb73abc1aeb866fb106"
Nov 02 13:15:04.797908 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.797892    5879 generic.go:356] "Generic (PLEG): container finished" podID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" containerID="dace30a2c4be706fd06c9340bdc1148da2ba0c3c44cdf759e084806b16347536" exitCode=60
Nov 02 13:15:04.797986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.797957    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt" event={"ID":"5ffec64c-ff1f-42df-83a2-3d45751f8bb7","Type":"ContainerDied","Data":"dace30a2c4be706fd06c9340bdc1148da2ba0c3c44cdf759e084806b16347536"}
Nov 02 13:15:04.798252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.798223    5879 scope.go:122] "RemoveContainer" containerID="dace30a2c4be706fd06c9340bdc1148da2ba0c3c44cdf759e084806b16347536"
Nov 02 13:15:04.799911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.799878    5879 generic.go:356] "Generic (PLEG): container finished" podID="e518dcfa-edd5-4c9c-b128-3504862ab477" containerID="88d3e058cafe80aa5d3068cc356467e335c12566e606752ad7ec904fdc9fef87" exitCode=60
Nov 02 13:15:04.800012 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.799952    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq" event={"ID":"e518dcfa-edd5-4c9c-b128-3504862ab477","Type":"ContainerDied","Data":"88d3e058cafe80aa5d3068cc356467e335c12566e606752ad7ec904fdc9fef87"}
Nov 02 13:15:04.800288 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.800271    5879 scope.go:122] "RemoveContainer" containerID="88d3e058cafe80aa5d3068cc356467e335c12566e606752ad7ec904fdc9fef87"
Nov 02 13:15:04.801869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.801848    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" event={"ID":"d8964b32-2de3-4bb6-b25c-7d7a561d0251","Type":"ContainerStarted","Data":"9236f16af156aab9f5e9ec174311154d242387f75a839d132b3fd3535d2feeca"}
Nov 02 13:15:04.803633 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.803612    5879 generic.go:356] "Generic (PLEG): container finished" podID="2ba4d701-2d78-40cf-bacc-8206cc16e923" containerID="359f829e2449079a7110f52e2a29e65401614c770e68ebe7e796f749cbe0edcb" exitCode=60
Nov 02 13:15:04.803749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.803686    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz" event={"ID":"2ba4d701-2d78-40cf-bacc-8206cc16e923","Type":"ContainerDied","Data":"359f829e2449079a7110f52e2a29e65401614c770e68ebe7e796f749cbe0edcb"}
Nov 02 13:15:04.803901 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.803884    5879 scope.go:122] "RemoveContainer" containerID="359f829e2449079a7110f52e2a29e65401614c770e68ebe7e796f749cbe0edcb"
Nov 02 13:15:04.805395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.805374    5879 generic.go:356] "Generic (PLEG): container finished" podID="20982710-74e3-4acb-af83-0f1d76338674" containerID="ee700de62756d55f9630a10bd780fcbb46a4d25dca96af269c951513b392de6f" exitCode=60
Nov 02 13:15:04.805481 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.805436    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg" event={"ID":"20982710-74e3-4acb-af83-0f1d76338674","Type":"ContainerDied","Data":"ee700de62756d55f9630a10bd780fcbb46a4d25dca96af269c951513b392de6f"}
Nov 02 13:15:04.805644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.805629    5879 scope.go:122] "RemoveContainer" containerID="ee700de62756d55f9630a10bd780fcbb46a4d25dca96af269c951513b392de6f"
Nov 02 13:15:04.812921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.812887    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78" event={"ID":"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05","Type":"ContainerStarted","Data":"9aa67b6906766663af66c57e47a3c7192f28644357d68f21a6ccb6ab0fcf56ee"}
Nov 02 13:15:04.815717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.815686    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt" event={"ID":"9025c2c7-57b9-439c-bbf8-91a5fd1433c0","Type":"ContainerStarted","Data":"29475aafe89f01897a06e69b8219499a293ecd1c52f3024611050b0d17795ae9"}
Nov 02 13:15:04.818311 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.818280    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv" event={"ID":"e9110a44-4129-4c65-9de1-f2574890c394","Type":"ContainerStarted","Data":"7d2bf8a34940df6420e67d44f5a132113bf18de7296e49d7947c0898a4edbf7f"}
Nov 02 13:15:04.827845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.827802    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2" event={"ID":"031e33ac-9ee4-4153-9bdb-5289b569a146","Type":"ContainerStarted","Data":"541e2630507f08aef75e621f0f828c1c7aacd335f2e001ee7c13af60b90345f8"}
Nov 02 13:15:04.832593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.832561    5879 generic.go:356] "Generic (PLEG): container finished" podID="021f486a-e073-4944-8b61-ecece24ff7fe" containerID="e45aabf958ccba68367978af9d41a7333b635dd72c00c613b8bce81560b5704c" exitCode=60
Nov 02 13:15:04.832915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.832844    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9" event={"ID":"021f486a-e073-4944-8b61-ecece24ff7fe","Type":"ContainerDied","Data":"e45aabf958ccba68367978af9d41a7333b635dd72c00c613b8bce81560b5704c"}
Nov 02 13:15:04.833767 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.833742    5879 scope.go:122] "RemoveContainer" containerID="e45aabf958ccba68367978af9d41a7333b635dd72c00c613b8bce81560b5704c"
Nov 02 13:15:04.839164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.839130    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd" event={"ID":"397a2f80-d67d-43d4-a4f8-35f6e93e111e","Type":"ContainerStarted","Data":"b3d27f5f0b68d3c236f68fab69ad9773f8c5e477c99a5c02869914668465d2b0"}
Nov 02 13:15:04.843314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.841983    5879 generic.go:356] "Generic (PLEG): container finished" podID="265f8052-98a4-44a8-95ef-239d37e81c43" containerID="e8eb5169a7c3f7fac80842c3d82d133e1760cbee5437502ddb40e5fd2da3ea82" exitCode=60
Nov 02 13:15:04.843314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.842152    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr" event={"ID":"265f8052-98a4-44a8-95ef-239d37e81c43","Type":"ContainerDied","Data":"e8eb5169a7c3f7fac80842c3d82d133e1760cbee5437502ddb40e5fd2da3ea82"}
Nov 02 13:15:04.843314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.842604    5879 scope.go:122] "RemoveContainer" containerID="e8eb5169a7c3f7fac80842c3d82d133e1760cbee5437502ddb40e5fd2da3ea82"
Nov 02 13:15:04.846487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.846457    5879 generic.go:356] "Generic (PLEG): container finished" podID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" containerID="75ec9c912b3d971cf143cdb649e0901ee0a227873551197ca97029634f7442de" exitCode=60
Nov 02 13:15:04.846696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.846679    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8" event={"ID":"c34bf9c9-3f3e-47d2-adaa-8f473dec78be","Type":"ContainerDied","Data":"75ec9c912b3d971cf143cdb649e0901ee0a227873551197ca97029634f7442de"}
Nov 02 13:15:04.847394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.847366    5879 scope.go:122] "RemoveContainer" containerID="75ec9c912b3d971cf143cdb649e0901ee0a227873551197ca97029634f7442de"
Nov 02 13:15:04.849506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.849481    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5" event={"ID":"23754db5-04c9-48a5-841a-433830943be2","Type":"ContainerStarted","Data":"d14433d33e683dd01b0b2617b8232dac23cdd6f90edd905d23c20158a9258a5e"}
Nov 02 13:15:04.856900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.855899    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82" podStartSLOduration=4.85588277 podStartE2EDuration="4.85588277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.845180858 +0000 UTC m=+2850.951883713" watchObservedRunningTime="2026-11-02 13:15:04.85588277 +0000 UTC m=+2850.962585622"
Nov 02 13:15:04.861343 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.861311    5879 generic.go:356] "Generic (PLEG): container finished" podID="4c723d82-c700-4def-b8c0-081c5e358b18" containerID="e91cb4eb4d0de76fdd5bb737d0202215e394a1a990e2a38f01ce89fbefb57286" exitCode=60
Nov 02 13:15:04.861467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.861362    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl" event={"ID":"4c723d82-c700-4def-b8c0-081c5e358b18","Type":"ContainerDied","Data":"e91cb4eb4d0de76fdd5bb737d0202215e394a1a990e2a38f01ce89fbefb57286"}
Nov 02 13:15:04.861748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.861727    5879 scope.go:122] "RemoveContainer" containerID="e91cb4eb4d0de76fdd5bb737d0202215e394a1a990e2a38f01ce89fbefb57286"
Nov 02 13:15:04.992135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:04.991932    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd" podStartSLOduration=4.991914294 podStartE2EDuration="4.991914294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:04.990736178 +0000 UTC m=+2851.097439029" watchObservedRunningTime="2026-11-02 13:15:04.991914294 +0000 UTC m=+2851.098617136"
Nov 02 13:15:05.039104 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.039049    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt" podStartSLOduration=5.039032431 podStartE2EDuration="5.039032431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:05.037379203 +0000 UTC m=+2851.144082054" watchObservedRunningTime="2026-11-02 13:15:05.039032431 +0000 UTC m=+2851.145735346"
Nov 02 13:15:05.255321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.255256    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2" podStartSLOduration=5.255238168 podStartE2EDuration="5.255238168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:05.252409261 +0000 UTC m=+2851.359112108" watchObservedRunningTime="2026-11-02 13:15:05.255238168 +0000 UTC m=+2851.361941019"
Nov 02 13:15:05.341400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.341226    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv" podStartSLOduration=5.341208206 podStartE2EDuration="5.341208206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:05.338192941 +0000 UTC m=+2851.444895792" watchObservedRunningTime="2026-11-02 13:15:05.341208206 +0000 UTC m=+2851.447911056"
Nov 02 13:15:05.567695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.567639    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" podStartSLOduration=5.567621853 podStartE2EDuration="5.567621853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:05.565146187 +0000 UTC m=+2851.671849038" watchObservedRunningTime="2026-11-02 13:15:05.567621853 +0000 UTC m=+2851.674324704"
Nov 02 13:15:05.581163 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.581122    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:05.670943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.670896    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5" podStartSLOduration=5.670882161 podStartE2EDuration="5.670882161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:15:05.66830997 +0000 UTC m=+2851.775012824" watchObservedRunningTime="2026-11-02 13:15:05.670882161 +0000 UTC m=+2851.777585043"
Nov 02 13:15:05.734674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.734611    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:05.901686 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.896131    5879 generic.go:356] "Generic (PLEG): container finished" podID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" containerID="9aa67b6906766663af66c57e47a3c7192f28644357d68f21a6ccb6ab0fcf56ee" exitCode=60
Nov 02 13:15:05.901686 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.896215    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78" event={"ID":"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05","Type":"ContainerDied","Data":"9aa67b6906766663af66c57e47a3c7192f28644357d68f21a6ccb6ab0fcf56ee"}
Nov 02 13:15:05.901686 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.896272    5879 scope.go:122] "RemoveContainer" containerID="b9ff7295da6964d240ec3d6fa72462f503584eddea6e494af5f91fd37d399884"
Nov 02 13:15:05.901686 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.896885    5879 scope.go:122] "RemoveContainer" containerID="9aa67b6906766663af66c57e47a3c7192f28644357d68f21a6ccb6ab0fcf56ee"
Nov 02 13:15:05.901686 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:05.897341    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78_odh-ai-gateway-infra(1e1b6178-a8b0-4b37-ae61-e2ce108e7a05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05"
Nov 02 13:15:05.912149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.911776    5879 generic.go:356] "Generic (PLEG): container finished" podID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" containerID="29475aafe89f01897a06e69b8219499a293ecd1c52f3024611050b0d17795ae9" exitCode=60
Nov 02 13:15:05.912149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.911889    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt" event={"ID":"9025c2c7-57b9-439c-bbf8-91a5fd1433c0","Type":"ContainerDied","Data":"29475aafe89f01897a06e69b8219499a293ecd1c52f3024611050b0d17795ae9"}
Nov 02 13:15:05.913230 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.912756    5879 scope.go:122] "RemoveContainer" containerID="29475aafe89f01897a06e69b8219499a293ecd1c52f3024611050b0d17795ae9"
Nov 02 13:15:05.938579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.938526    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:05.977414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.960319    5879 generic.go:356] "Generic (PLEG): container finished" podID="e9110a44-4129-4c65-9de1-f2574890c394" containerID="7d2bf8a34940df6420e67d44f5a132113bf18de7296e49d7947c0898a4edbf7f" exitCode=60
Nov 02 13:15:05.977619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.960367    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv" event={"ID":"e9110a44-4129-4c65-9de1-f2574890c394","Type":"ContainerDied","Data":"7d2bf8a34940df6420e67d44f5a132113bf18de7296e49d7947c0898a4edbf7f"}
Nov 02 13:15:05.977619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.976776    5879 scope.go:122] "RemoveContainer" containerID="7d2bf8a34940df6420e67d44f5a132113bf18de7296e49d7947c0898a4edbf7f"
Nov 02 13:15:05.980953 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.980924    5879 generic.go:356] "Generic (PLEG): container finished" podID="031e33ac-9ee4-4153-9bdb-5289b569a146" containerID="541e2630507f08aef75e621f0f828c1c7aacd335f2e001ee7c13af60b90345f8" exitCode=60
Nov 02 13:15:05.989881 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.989822    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2" event={"ID":"031e33ac-9ee4-4153-9bdb-5289b569a146","Type":"ContainerDied","Data":"541e2630507f08aef75e621f0f828c1c7aacd335f2e001ee7c13af60b90345f8"}
Nov 02 13:15:05.992035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:05.991709    5879 scope.go:122] "RemoveContainer" containerID="541e2630507f08aef75e621f0f828c1c7aacd335f2e001ee7c13af60b90345f8"
Nov 02 13:15:06.006084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:06.001504    5879 generic.go:356] "Generic (PLEG): container finished" podID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" containerID="b3d27f5f0b68d3c236f68fab69ad9773f8c5e477c99a5c02869914668465d2b0" exitCode=60
Nov 02 13:15:06.006084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:06.002170    5879 scope.go:122] "RemoveContainer" containerID="b3d27f5f0b68d3c236f68fab69ad9773f8c5e477c99a5c02869914668465d2b0"
Nov 02 13:15:06.006084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:06.002212    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd" event={"ID":"397a2f80-d67d-43d4-a4f8-35f6e93e111e","Type":"ContainerDied","Data":"b3d27f5f0b68d3c236f68fab69ad9773f8c5e477c99a5c02869914668465d2b0"}
Nov 02 13:15:06.007814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:06.007787    5879 generic.go:356] "Generic (PLEG): container finished" podID="23754db5-04c9-48a5-841a-433830943be2" containerID="d14433d33e683dd01b0b2617b8232dac23cdd6f90edd905d23c20158a9258a5e" exitCode=60
Nov 02 13:15:06.008142 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:06.008120    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5" event={"ID":"23754db5-04c9-48a5-841a-433830943be2","Type":"ContainerDied","Data":"d14433d33e683dd01b0b2617b8232dac23cdd6f90edd905d23c20158a9258a5e"}
Nov 02 13:15:06.008984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:06.008965    5879 scope.go:122] "RemoveContainer" containerID="d14433d33e683dd01b0b2617b8232dac23cdd6f90edd905d23c20158a9258a5e"
Nov 02 13:15:06.034497 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:06.034470    5879 generic.go:356] "Generic (PLEG): container finished" podID="e97847d4-325d-4ce4-874d-199279b16a87" containerID="7b93b154c5b0c02c714ed5901151c4e5d5b17e416e9ea9d82bb726d25168bc68" exitCode=60
Nov 02 13:15:06.035637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:06.035603    5879 scope.go:122] "RemoveContainer" containerID="7b93b154c5b0c02c714ed5901151c4e5d5b17e416e9ea9d82bb726d25168bc68"
Nov 02 13:15:06.037181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:06.037154    5879 generic.go:356] "Generic (PLEG): container finished" podID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" containerID="9236f16af156aab9f5e9ec174311154d242387f75a839d132b3fd3535d2feeca" exitCode=60
Nov 02 13:15:06.037831 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:06.037653    5879 scope.go:122] "RemoveContainer" containerID="9236f16af156aab9f5e9ec174311154d242387f75a839d132b3fd3535d2feeca"
Nov 02 13:15:06.041870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:06.041828    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82" event={"ID":"e97847d4-325d-4ce4-874d-199279b16a87","Type":"ContainerDied","Data":"7b93b154c5b0c02c714ed5901151c4e5d5b17e416e9ea9d82bb726d25168bc68"}
Nov 02 13:15:06.041979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:06.041909    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" event={"ID":"d8964b32-2de3-4bb6-b25c-7d7a561d0251","Type":"ContainerDied","Data":"9236f16af156aab9f5e9ec174311154d242387f75a839d132b3fd3535d2feeca"}
Nov 02 13:15:07.051702 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.051603    5879 generic.go:356] "Generic (PLEG): container finished" podID="2ba4d701-2d78-40cf-bacc-8206cc16e923" containerID="f6239f5ca272786738d1a3dc16ea806998f2ed8036e98833c0558cd23ff40579" exitCode=60
Nov 02 13:15:07.051702 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.051687    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz" event={"ID":"2ba4d701-2d78-40cf-bacc-8206cc16e923","Type":"ContainerDied","Data":"f6239f5ca272786738d1a3dc16ea806998f2ed8036e98833c0558cd23ff40579"}
Nov 02 13:15:07.052265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.051753    5879 scope.go:122] "RemoveContainer" containerID="359f829e2449079a7110f52e2a29e65401614c770e68ebe7e796f749cbe0edcb"
Nov 02 13:15:07.052265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.052192    5879 scope.go:122] "RemoveContainer" containerID="f6239f5ca272786738d1a3dc16ea806998f2ed8036e98833c0558cd23ff40579"
Nov 02 13:15:07.052646 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:07.052620    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz_odh-ai-gateway-infra(2ba4d701-2d78-40cf-bacc-8206cc16e923)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923"
Nov 02 13:15:07.054399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.054376    5879 generic.go:356] "Generic (PLEG): container finished" podID="20982710-74e3-4acb-af83-0f1d76338674" containerID="e608a17961c1da22e6ceedc10051cc61bb1b41496177cf4dbdd7ed25d18c9709" exitCode=60
Nov 02 13:15:07.054518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.054475    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg" event={"ID":"20982710-74e3-4acb-af83-0f1d76338674","Type":"ContainerDied","Data":"e608a17961c1da22e6ceedc10051cc61bb1b41496177cf4dbdd7ed25d18c9709"}
Nov 02 13:15:07.054836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.054819    5879 scope.go:122] "RemoveContainer" containerID="e608a17961c1da22e6ceedc10051cc61bb1b41496177cf4dbdd7ed25d18c9709"
Nov 02 13:15:07.055335 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:07.055305    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg_odh-ai-gateway-infra(20982710-74e3-4acb-af83-0f1d76338674)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg" podUID="20982710-74e3-4acb-af83-0f1d76338674"
Nov 02 13:15:07.057308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.057174    5879 scope.go:122] "RemoveContainer" containerID="9aa67b6906766663af66c57e47a3c7192f28644357d68f21a6ccb6ab0fcf56ee"
Nov 02 13:15:07.057659 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:07.057633    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78_odh-ai-gateway-infra(1e1b6178-a8b0-4b37-ae61-e2ce108e7a05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05"
Nov 02 13:15:07.059899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.059871    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt" event={"ID":"9025c2c7-57b9-439c-bbf8-91a5fd1433c0","Type":"ContainerStarted","Data":"0b743614ff0627827cc4143ee4ff03e456e1cf1d4f4b3cce6beb33e9975964e9"}
Nov 02 13:15:07.062354 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.062324    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2" event={"ID":"031e33ac-9ee4-4153-9bdb-5289b569a146","Type":"ContainerStarted","Data":"fe5670dcd2998a923d073f1e7ce62be5a61544ae034fb7e6280627541067182d"}
Nov 02 13:15:07.064728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.064701    5879 generic.go:356] "Generic (PLEG): container finished" podID="021f486a-e073-4944-8b61-ecece24ff7fe" containerID="f9432cf86284898c32e0cf65d24e012d7b215cf37a6ffeff9f76a9683f6c3d4e" exitCode=60
Nov 02 13:15:07.064846 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.064744    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9" event={"ID":"021f486a-e073-4944-8b61-ecece24ff7fe","Type":"ContainerDied","Data":"f9432cf86284898c32e0cf65d24e012d7b215cf37a6ffeff9f76a9683f6c3d4e"}
Nov 02 13:15:07.065093 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.065079    5879 scope.go:122] "RemoveContainer" containerID="f9432cf86284898c32e0cf65d24e012d7b215cf37a6ffeff9f76a9683f6c3d4e"
Nov 02 13:15:07.065450 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:07.065419    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9_odh-ai-gateway-infra(021f486a-e073-4944-8b61-ecece24ff7fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9" podUID="021f486a-e073-4944-8b61-ecece24ff7fe"
Nov 02 13:15:07.066958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.066936    5879 generic.go:356] "Generic (PLEG): container finished" podID="265f8052-98a4-44a8-95ef-239d37e81c43" containerID="2c98a988d37251c0732ad78b772301f400f6bc777f7f93f0e32b743461df4f3b" exitCode=60
Nov 02 13:15:07.067181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.067018    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr" event={"ID":"265f8052-98a4-44a8-95ef-239d37e81c43","Type":"ContainerDied","Data":"2c98a988d37251c0732ad78b772301f400f6bc777f7f93f0e32b743461df4f3b"}
Nov 02 13:15:07.067357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.067340    5879 scope.go:122] "RemoveContainer" containerID="2c98a988d37251c0732ad78b772301f400f6bc777f7f93f0e32b743461df4f3b"
Nov 02 13:15:07.067704 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:07.067684    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr_odh-ai-gateway-infra(265f8052-98a4-44a8-95ef-239d37e81c43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr" podUID="265f8052-98a4-44a8-95ef-239d37e81c43"
Nov 02 13:15:07.069225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.069205    5879 generic.go:356] "Generic (PLEG): container finished" podID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" containerID="d5d0d54313768f763250b8ee0be59416626ae75ab10e88f01f2a41187c9404da" exitCode=60
Nov 02 13:15:07.069338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.069257    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8" event={"ID":"c34bf9c9-3f3e-47d2-adaa-8f473dec78be","Type":"ContainerDied","Data":"d5d0d54313768f763250b8ee0be59416626ae75ab10e88f01f2a41187c9404da"}
Nov 02 13:15:07.069588 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.069570    5879 scope.go:122] "RemoveContainer" containerID="d5d0d54313768f763250b8ee0be59416626ae75ab10e88f01f2a41187c9404da"
Nov 02 13:15:07.069931 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:07.069903    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8_odh-ai-gateway-infra(c34bf9c9-3f3e-47d2-adaa-8f473dec78be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be"
Nov 02 13:15:07.071573 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.071549    5879 generic.go:356] "Generic (PLEG): container finished" podID="4c723d82-c700-4def-b8c0-081c5e358b18" containerID="5de7a9251738ba2a143a64c1d6b95b865c4d8a2824b6b4fa12ef3174b221702e" exitCode=60
Nov 02 13:15:07.071668 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.071650    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl" event={"ID":"4c723d82-c700-4def-b8c0-081c5e358b18","Type":"ContainerDied","Data":"5de7a9251738ba2a143a64c1d6b95b865c4d8a2824b6b4fa12ef3174b221702e"}
Nov 02 13:15:07.071903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.071887    5879 scope.go:122] "RemoveContainer" containerID="5de7a9251738ba2a143a64c1d6b95b865c4d8a2824b6b4fa12ef3174b221702e"
Nov 02 13:15:07.072283 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:07.072257    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl_odh-ai-gateway-infra(4c723d82-c700-4def-b8c0-081c5e358b18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl" podUID="4c723d82-c700-4def-b8c0-081c5e358b18"
Nov 02 13:15:07.074021 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.073982    5879 generic.go:356] "Generic (PLEG): container finished" podID="4863e18b-0942-4e01-94e1-293fd89aa56f" containerID="00bece299034862b1cd2fedca7541f12495be204d8c73d6bfd2c06ac63e4565b" exitCode=60
Nov 02 13:15:07.074127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.074019    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw" event={"ID":"4863e18b-0942-4e01-94e1-293fd89aa56f","Type":"ContainerDied","Data":"00bece299034862b1cd2fedca7541f12495be204d8c73d6bfd2c06ac63e4565b"}
Nov 02 13:15:07.074372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.074353    5879 scope.go:122] "RemoveContainer" containerID="00bece299034862b1cd2fedca7541f12495be204d8c73d6bfd2c06ac63e4565b"
Nov 02 13:15:07.074730 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:07.074699    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw_odh-ai-gateway-infra(4863e18b-0942-4e01-94e1-293fd89aa56f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f"
Nov 02 13:15:07.076250 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.076230    5879 generic.go:356] "Generic (PLEG): container finished" podID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" containerID="d118e345741850bbfdd3f629b4b3b2ccc8aa74a00c0d78fcf06201a5899b4aef" exitCode=60
Nov 02 13:15:07.076345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.076318    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk" event={"ID":"9262bfd3-6bbc-4161-afa6-93fd1a6639a1","Type":"ContainerDied","Data":"d118e345741850bbfdd3f629b4b3b2ccc8aa74a00c0d78fcf06201a5899b4aef"}
Nov 02 13:15:07.076625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.076610    5879 scope.go:122] "RemoveContainer" containerID="d118e345741850bbfdd3f629b4b3b2ccc8aa74a00c0d78fcf06201a5899b4aef"
Nov 02 13:15:07.076956 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:07.076933    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk_odh-ai-gateway-infra(9262bfd3-6bbc-4161-afa6-93fd1a6639a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1"
Nov 02 13:15:07.078493 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.078471    5879 generic.go:356] "Generic (PLEG): container finished" podID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" containerID="8d887314fa0303726d5575e93d25ae097080e77e8d1a9d7e0a99197a2482c432" exitCode=60
Nov 02 13:15:07.078602 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.078549    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt" event={"ID":"5ffec64c-ff1f-42df-83a2-3d45751f8bb7","Type":"ContainerDied","Data":"8d887314fa0303726d5575e93d25ae097080e77e8d1a9d7e0a99197a2482c432"}
Nov 02 13:15:07.078788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.078769    5879 scope.go:122] "RemoveContainer" containerID="8d887314fa0303726d5575e93d25ae097080e77e8d1a9d7e0a99197a2482c432"
Nov 02 13:15:07.079071 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:07.079050    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt_odh-ai-gateway-infra(5ffec64c-ff1f-42df-83a2-3d45751f8bb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7"
Nov 02 13:15:07.080857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.080834    5879 generic.go:356] "Generic (PLEG): container finished" podID="e518dcfa-edd5-4c9c-b128-3504862ab477" containerID="ce2e879f6d790f77b9b332de995435fbf9fe8f4ec1ebfca455734e8927e0aae1" exitCode=60
Nov 02 13:15:07.080954 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.080914    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq" event={"ID":"e518dcfa-edd5-4c9c-b128-3504862ab477","Type":"ContainerDied","Data":"ce2e879f6d790f77b9b332de995435fbf9fe8f4ec1ebfca455734e8927e0aae1"}
Nov 02 13:15:07.081218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.081204    5879 scope.go:122] "RemoveContainer" containerID="ce2e879f6d790f77b9b332de995435fbf9fe8f4ec1ebfca455734e8927e0aae1"
Nov 02 13:15:07.081485 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:07.081469    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq_odh-ai-gateway-infra(e518dcfa-edd5-4c9c-b128-3504862ab477)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477"
Nov 02 13:15:07.374119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.374044    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:07.412473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.411508    5879 scope.go:122] "RemoveContainer" containerID="ee700de62756d55f9630a10bd780fcbb46a4d25dca96af269c951513b392de6f"
Nov 02 13:15:07.496547 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.496517    5879 scope.go:122] "RemoveContainer" containerID="e45aabf958ccba68367978af9d41a7333b635dd72c00c613b8bce81560b5704c"
Nov 02 13:15:07.503453 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.503403    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:07.535191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.535168    5879 scope.go:122] "RemoveContainer" containerID="e8eb5169a7c3f7fac80842c3d82d133e1760cbee5437502ddb40e5fd2da3ea82"
Nov 02 13:15:07.552985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.552957    5879 scope.go:122] "RemoveContainer" containerID="75ec9c912b3d971cf143cdb649e0901ee0a227873551197ca97029634f7442de"
Nov 02 13:15:07.575967 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.575938    5879 scope.go:122] "RemoveContainer" containerID="e91cb4eb4d0de76fdd5bb737d0202215e394a1a990e2a38f01ce89fbefb57286"
Nov 02 13:15:07.601105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.600744    5879 scope.go:122] "RemoveContainer" containerID="9366f1256db9bced1a747174c1fba36230480843e5f5796b5de294293d0d7158"
Nov 02 13:15:07.814530 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.814474    5879 scope.go:122] "RemoveContainer" containerID="8ac42d71b010e80747a49587decbaf98fe535b6570a75cb73abc1aeb866fb106"
Nov 02 13:15:07.854606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.854580    5879 scope.go:122] "RemoveContainer" containerID="dace30a2c4be706fd06c9340bdc1148da2ba0c3c44cdf759e084806b16347536"
Nov 02 13:15:07.874207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:07.874174    5879 scope.go:122] "RemoveContainer" containerID="88d3e058cafe80aa5d3068cc356467e335c12566e606752ad7ec904fdc9fef87"
Nov 02 13:15:08.115192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.115146    5879 generic.go:356] "Generic (PLEG): container finished" podID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" containerID="0b743614ff0627827cc4143ee4ff03e456e1cf1d4f4b3cce6beb33e9975964e9" exitCode=60
Nov 02 13:15:08.116084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.115237    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt" event={"ID":"9025c2c7-57b9-439c-bbf8-91a5fd1433c0","Type":"ContainerDied","Data":"0b743614ff0627827cc4143ee4ff03e456e1cf1d4f4b3cce6beb33e9975964e9"}
Nov 02 13:15:08.116084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.115346    5879 scope.go:122] "RemoveContainer" containerID="29475aafe89f01897a06e69b8219499a293ecd1c52f3024611050b0d17795ae9"
Nov 02 13:15:08.116084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.115613    5879 scope.go:122] "RemoveContainer" containerID="0b743614ff0627827cc4143ee4ff03e456e1cf1d4f4b3cce6beb33e9975964e9"
Nov 02 13:15:08.116285 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:08.116086    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt_odh-ai-gateway-infra(9025c2c7-57b9-439c-bbf8-91a5fd1433c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0"
Nov 02 13:15:08.119056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.119031    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv" event={"ID":"e9110a44-4129-4c65-9de1-f2574890c394","Type":"ContainerStarted","Data":"c128541ac14a5f8ea7083fc905f45f761544966546093c827b4060d086111960"}
Nov 02 13:15:08.121847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.121821    5879 generic.go:356] "Generic (PLEG): container finished" podID="031e33ac-9ee4-4153-9bdb-5289b569a146" containerID="fe5670dcd2998a923d073f1e7ce62be5a61544ae034fb7e6280627541067182d" exitCode=60
Nov 02 13:15:08.121970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.121925    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2" event={"ID":"031e33ac-9ee4-4153-9bdb-5289b569a146","Type":"ContainerDied","Data":"fe5670dcd2998a923d073f1e7ce62be5a61544ae034fb7e6280627541067182d"}
Nov 02 13:15:08.122278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.122260    5879 scope.go:122] "RemoveContainer" containerID="fe5670dcd2998a923d073f1e7ce62be5a61544ae034fb7e6280627541067182d"
Nov 02 13:15:08.122715 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:08.122682    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2_odh-ai-gateway-infra(031e33ac-9ee4-4153-9bdb-5289b569a146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146"
Nov 02 13:15:08.124760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.124742    5879 scope.go:122] "RemoveContainer" containerID="f9432cf86284898c32e0cf65d24e012d7b215cf37a6ffeff9f76a9683f6c3d4e"
Nov 02 13:15:08.125220 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:08.125184    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9_odh-ai-gateway-infra(021f486a-e073-4944-8b61-ecece24ff7fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9" podUID="021f486a-e073-4944-8b61-ecece24ff7fe"
Nov 02 13:15:08.127157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.127111    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd" event={"ID":"397a2f80-d67d-43d4-a4f8-35f6e93e111e","Type":"ContainerStarted","Data":"a5cea5ddb71df623f6b5031b6e3ff2d340a33a8b68c344b39a8c8a09f4d9c5e2"}
Nov 02 13:15:08.129810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.129788    5879 scope.go:122] "RemoveContainer" containerID="2c98a988d37251c0732ad78b772301f400f6bc777f7f93f0e32b743461df4f3b"
Nov 02 13:15:08.130197 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:08.130175    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr_odh-ai-gateway-infra(265f8052-98a4-44a8-95ef-239d37e81c43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr" podUID="265f8052-98a4-44a8-95ef-239d37e81c43"
Nov 02 13:15:08.132109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.132088    5879 scope.go:122] "RemoveContainer" containerID="d5d0d54313768f763250b8ee0be59416626ae75ab10e88f01f2a41187c9404da"
Nov 02 13:15:08.132483 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:08.132452    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8_odh-ai-gateway-infra(c34bf9c9-3f3e-47d2-adaa-8f473dec78be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be"
Nov 02 13:15:08.134195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.134168    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5" event={"ID":"23754db5-04c9-48a5-841a-433830943be2","Type":"ContainerStarted","Data":"7783195e1d7939582645e7daf634638826e2aa5d7111433df53e937abd641c44"}
Nov 02 13:15:08.136096 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.136077    5879 scope.go:122] "RemoveContainer" containerID="541e2630507f08aef75e621f0f828c1c7aacd335f2e001ee7c13af60b90345f8"
Nov 02 13:15:08.136445 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.136427    5879 scope.go:122] "RemoveContainer" containerID="5de7a9251738ba2a143a64c1d6b95b865c4d8a2824b6b4fa12ef3174b221702e"
Nov 02 13:15:08.136714 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:08.136694    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl_odh-ai-gateway-infra(4c723d82-c700-4def-b8c0-081c5e358b18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl" podUID="4c723d82-c700-4def-b8c0-081c5e358b18"
Nov 02 13:15:08.138802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.138778    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82" event={"ID":"e97847d4-325d-4ce4-874d-199279b16a87","Type":"ContainerStarted","Data":"9bd72b084452751467823ba6e1f1cf8f3ff540777714df46d2306e6a708a03be"}
Nov 02 13:15:08.141462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.141443    5879 scope.go:122] "RemoveContainer" containerID="00bece299034862b1cd2fedca7541f12495be204d8c73d6bfd2c06ac63e4565b"
Nov 02 13:15:08.141770 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:08.141749    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw_odh-ai-gateway-infra(4863e18b-0942-4e01-94e1-293fd89aa56f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f"
Nov 02 13:15:08.143664 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.143644    5879 scope.go:122] "RemoveContainer" containerID="d118e345741850bbfdd3f629b4b3b2ccc8aa74a00c0d78fcf06201a5899b4aef"
Nov 02 13:15:08.143930 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:08.143909    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk_odh-ai-gateway-infra(9262bfd3-6bbc-4161-afa6-93fd1a6639a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1"
Nov 02 13:15:08.145495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.145479    5879 scope.go:122] "RemoveContainer" containerID="8d887314fa0303726d5575e93d25ae097080e77e8d1a9d7e0a99197a2482c432"
Nov 02 13:15:08.145750 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:08.145725    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt_odh-ai-gateway-infra(5ffec64c-ff1f-42df-83a2-3d45751f8bb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7"
Nov 02 13:15:08.147341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.147322    5879 scope.go:122] "RemoveContainer" containerID="ce2e879f6d790f77b9b332de995435fbf9fe8f4ec1ebfca455734e8927e0aae1"
Nov 02 13:15:08.147589 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:08.147569    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq_odh-ai-gateway-infra(e518dcfa-edd5-4c9c-b128-3504862ab477)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477"
Nov 02 13:15:08.148886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.148863    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:08.149413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.149395    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" event={"ID":"d8964b32-2de3-4bb6-b25c-7d7a561d0251","Type":"ContainerStarted","Data":"8403eab313e8742ac0d0737a691aaf555c192b2c4bd8ff7aa66f3882a08a18ae"}
Nov 02 13:15:08.151589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.151570    5879 scope.go:122] "RemoveContainer" containerID="f6239f5ca272786738d1a3dc16ea806998f2ed8036e98833c0558cd23ff40579"
Nov 02 13:15:08.151825 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:08.151807    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz_odh-ai-gateway-infra(2ba4d701-2d78-40cf-bacc-8206cc16e923)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923"
Nov 02 13:15:08.153436 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:08.153419    5879 scope.go:122] "RemoveContainer" containerID="e608a17961c1da22e6ceedc10051cc61bb1b41496177cf4dbdd7ed25d18c9709"
Nov 02 13:15:08.153634 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:08.153617    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg_odh-ai-gateway-infra(20982710-74e3-4acb-af83-0f1d76338674)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg" podUID="20982710-74e3-4acb-af83-0f1d76338674"
Nov 02 13:15:09.166283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.166250    5879 scope.go:122] "RemoveContainer" containerID="0b743614ff0627827cc4143ee4ff03e456e1cf1d4f4b3cce6beb33e9975964e9"
Nov 02 13:15:09.166780 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:09.166554    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt_odh-ai-gateway-infra(9025c2c7-57b9-439c-bbf8-91a5fd1433c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0"
Nov 02 13:15:09.168448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.168421    5879 generic.go:356] "Generic (PLEG): container finished" podID="e9110a44-4129-4c65-9de1-f2574890c394" containerID="c128541ac14a5f8ea7083fc905f45f761544966546093c827b4060d086111960" exitCode=60
Nov 02 13:15:09.168577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.168499    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv" event={"ID":"e9110a44-4129-4c65-9de1-f2574890c394","Type":"ContainerDied","Data":"c128541ac14a5f8ea7083fc905f45f761544966546093c827b4060d086111960"}
Nov 02 13:15:09.168577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.168567    5879 scope.go:122] "RemoveContainer" containerID="7d2bf8a34940df6420e67d44f5a132113bf18de7296e49d7947c0898a4edbf7f"
Nov 02 13:15:09.168735 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.168720    5879 scope.go:122] "RemoveContainer" containerID="c128541ac14a5f8ea7083fc905f45f761544966546093c827b4060d086111960"
Nov 02 13:15:09.169033 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:09.169013    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv_odh-ai-gateway-infra(e9110a44-4129-4c65-9de1-f2574890c394)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv" podUID="e9110a44-4129-4c65-9de1-f2574890c394"
Nov 02 13:15:09.170982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.170966    5879 scope.go:122] "RemoveContainer" containerID="fe5670dcd2998a923d073f1e7ce62be5a61544ae034fb7e6280627541067182d"
Nov 02 13:15:09.171309 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:09.171286    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2_odh-ai-gateway-infra(031e33ac-9ee4-4153-9bdb-5289b569a146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146"
Nov 02 13:15:09.172878 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.172857    5879 generic.go:356] "Generic (PLEG): container finished" podID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" containerID="a5cea5ddb71df623f6b5031b6e3ff2d340a33a8b68c344b39a8c8a09f4d9c5e2" exitCode=60
Nov 02 13:15:09.172981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.172933    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd" event={"ID":"397a2f80-d67d-43d4-a4f8-35f6e93e111e","Type":"ContainerDied","Data":"a5cea5ddb71df623f6b5031b6e3ff2d340a33a8b68c344b39a8c8a09f4d9c5e2"}
Nov 02 13:15:09.173184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.173163    5879 scope.go:122] "RemoveContainer" containerID="a5cea5ddb71df623f6b5031b6e3ff2d340a33a8b68c344b39a8c8a09f4d9c5e2"
Nov 02 13:15:09.173471 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:09.173445    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd_odh-ai-gateway-infra(397a2f80-d67d-43d4-a4f8-35f6e93e111e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e"
Nov 02 13:15:09.175083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.175062    5879 generic.go:356] "Generic (PLEG): container finished" podID="23754db5-04c9-48a5-841a-433830943be2" containerID="7783195e1d7939582645e7daf634638826e2aa5d7111433df53e937abd641c44" exitCode=60
Nov 02 13:15:09.175184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.175170    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5" event={"ID":"23754db5-04c9-48a5-841a-433830943be2","Type":"ContainerDied","Data":"7783195e1d7939582645e7daf634638826e2aa5d7111433df53e937abd641c44"}
Nov 02 13:15:09.175459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.175437    5879 scope.go:122] "RemoveContainer" containerID="7783195e1d7939582645e7daf634638826e2aa5d7111433df53e937abd641c44"
Nov 02 13:15:09.175847 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:09.175816    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5_odh-ai-gateway-infra(23754db5-04c9-48a5-841a-433830943be2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5" podUID="23754db5-04c9-48a5-841a-433830943be2"
Nov 02 13:15:09.177637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.177613    5879 generic.go:356] "Generic (PLEG): container finished" podID="e97847d4-325d-4ce4-874d-199279b16a87" containerID="9bd72b084452751467823ba6e1f1cf8f3ff540777714df46d2306e6a708a03be" exitCode=60
Nov 02 13:15:09.177784 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.177762    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82" event={"ID":"e97847d4-325d-4ce4-874d-199279b16a87","Type":"ContainerDied","Data":"9bd72b084452751467823ba6e1f1cf8f3ff540777714df46d2306e6a708a03be"}
Nov 02 13:15:09.177964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.177945    5879 scope.go:122] "RemoveContainer" containerID="9bd72b084452751467823ba6e1f1cf8f3ff540777714df46d2306e6a708a03be"
Nov 02 13:15:09.178315 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:09.178289    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82_odh-ai-gateway-infra(e97847d4-325d-4ce4-874d-199279b16a87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82" podUID="e97847d4-325d-4ce4-874d-199279b16a87"
Nov 02 13:15:09.180186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.180163    5879 generic.go:356] "Generic (PLEG): container finished" podID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" containerID="8403eab313e8742ac0d0737a691aaf555c192b2c4bd8ff7aa66f3882a08a18ae" exitCode=60
Nov 02 13:15:09.180277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.180211    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" event={"ID":"d8964b32-2de3-4bb6-b25c-7d7a561d0251","Type":"ContainerDied","Data":"8403eab313e8742ac0d0737a691aaf555c192b2c4bd8ff7aa66f3882a08a18ae"}
Nov 02 13:15:09.180475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.180461    5879 scope.go:122] "RemoveContainer" containerID="8403eab313e8742ac0d0737a691aaf555c192b2c4bd8ff7aa66f3882a08a18ae"
Nov 02 13:15:09.181401 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:09.181374    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c_odh-ai-gateway-infra(d8964b32-2de3-4bb6-b25c-7d7a561d0251)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251"
Nov 02 13:15:09.187823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.187805    5879 scope.go:122] "RemoveContainer" containerID="b3d27f5f0b68d3c236f68fab69ad9773f8c5e477c99a5c02869914668465d2b0"
Nov 02 13:15:09.203424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.203399    5879 scope.go:122] "RemoveContainer" containerID="d14433d33e683dd01b0b2617b8232dac23cdd6f90edd905d23c20158a9258a5e"
Nov 02 13:15:09.219570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.219550    5879 scope.go:122] "RemoveContainer" containerID="7b93b154c5b0c02c714ed5901151c4e5d5b17e416e9ea9d82bb726d25168bc68"
Nov 02 13:15:09.234568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.234545    5879 scope.go:122] "RemoveContainer" containerID="9236f16af156aab9f5e9ec174311154d242387f75a839d132b3fd3535d2feeca"
Nov 02 13:15:09.952290 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:09.952258    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:10.040632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:10.040589    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"]
Nov 02 13:15:10.045848 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:10.045821    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-512c850a-7ll95"]
Nov 02 13:15:10.131327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:10.131291    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:15:10.145887 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:10.145857    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:15:10.192688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:10.192650    5879 scope.go:122] "RemoveContainer" containerID="7783195e1d7939582645e7daf634638826e2aa5d7111433df53e937abd641c44"
Nov 02 13:15:10.193166 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:10.193083    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5_odh-ai-gateway-infra(23754db5-04c9-48a5-841a-433830943be2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5" podUID="23754db5-04c9-48a5-841a-433830943be2"
Nov 02 13:15:10.194501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:10.194480    5879 scope.go:122] "RemoveContainer" containerID="9bd72b084452751467823ba6e1f1cf8f3ff540777714df46d2306e6a708a03be"
Nov 02 13:15:10.194878 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:10.194857    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82_odh-ai-gateway-infra(e97847d4-325d-4ce4-874d-199279b16a87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82" podUID="e97847d4-325d-4ce4-874d-199279b16a87"
Nov 02 13:15:10.196689 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:10.196669    5879 scope.go:122] "RemoveContainer" containerID="8403eab313e8742ac0d0737a691aaf555c192b2c4bd8ff7aa66f3882a08a18ae"
Nov 02 13:15:10.197090 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:10.197061    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c_odh-ai-gateway-infra(d8964b32-2de3-4bb6-b25c-7d7a561d0251)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251"
Nov 02 13:15:10.198790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:10.198768    5879 scope.go:122] "RemoveContainer" containerID="c128541ac14a5f8ea7083fc905f45f761544966546093c827b4060d086111960"
Nov 02 13:15:10.199159 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:10.199142    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv_odh-ai-gateway-infra(e9110a44-4129-4c65-9de1-f2574890c394)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv" podUID="e9110a44-4129-4c65-9de1-f2574890c394"
Nov 02 13:15:10.200726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:10.200703    5879 scope.go:122] "RemoveContainer" containerID="a5cea5ddb71df623f6b5031b6e3ff2d340a33a8b68c344b39a8c8a09f4d9c5e2"
Nov 02 13:15:10.201147 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:10.201121    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd_odh-ai-gateway-infra(397a2f80-d67d-43d4-a4f8-35f6e93e111e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e"
Nov 02 13:15:10.211891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:10.211825    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:15:11.567983 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:11.567944    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:12.020186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:12.020151    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0515ea2-a0ef-408b-a552-025591ee29de" path="/var/lib/kubelet/pods/e0515ea2-a0ef-408b-a552-025591ee29de/volumes"
Nov 02 13:15:13.263501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:13.263460    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:14.823640 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:14.823606    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:15:14.835726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:14.835697    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:16.009955 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:16.009914    5879 scope.go:122] "RemoveContainer" containerID="9aa67b6906766663af66c57e47a3c7192f28644357d68f21a6ccb6ab0fcf56ee"
Nov 02 13:15:16.475790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:16.475755    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:17.010227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:17.010195    5879 scope.go:122] "RemoveContainer" containerID="d118e345741850bbfdd3f629b4b3b2ccc8aa74a00c0d78fcf06201a5899b4aef"
Nov 02 13:15:17.282240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:17.282203    5879 generic.go:356] "Generic (PLEG): container finished" podID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" containerID="307e5d98c56ce476947cabf5dc1877a35dc4b52084516d909b5a147ee7cb5fad" exitCode=60
Nov 02 13:15:17.282428 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:17.282277    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78" event={"ID":"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05","Type":"ContainerDied","Data":"307e5d98c56ce476947cabf5dc1877a35dc4b52084516d909b5a147ee7cb5fad"}
Nov 02 13:15:17.282428 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:17.282377    5879 scope.go:122] "RemoveContainer" containerID="9aa67b6906766663af66c57e47a3c7192f28644357d68f21a6ccb6ab0fcf56ee"
Nov 02 13:15:17.282824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:17.282798    5879 scope.go:122] "RemoveContainer" containerID="307e5d98c56ce476947cabf5dc1877a35dc4b52084516d909b5a147ee7cb5fad"
Nov 02 13:15:17.283242 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:17.283213    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78_odh-ai-gateway-infra(1e1b6178-a8b0-4b37-ae61-e2ce108e7a05)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05"
Nov 02 13:15:18.009910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.009885    5879 scope.go:122] "RemoveContainer" containerID="ce2e879f6d790f77b9b332de995435fbf9fe8f4ec1ebfca455734e8927e0aae1"
Nov 02 13:15:18.010290 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.010268    5879 scope.go:122] "RemoveContainer" containerID="8d887314fa0303726d5575e93d25ae097080e77e8d1a9d7e0a99197a2482c432"
Nov 02 13:15:18.010660 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.010421    5879 scope.go:122] "RemoveContainer" containerID="5de7a9251738ba2a143a64c1d6b95b865c4d8a2824b6b4fa12ef3174b221702e"
Nov 02 13:15:18.151442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.151410    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:18.295364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.295272    5879 generic.go:356] "Generic (PLEG): container finished" podID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" containerID="d710cb2ce57053310c79e8dc8b51bfc2b953fc97096897e9da9b760c45e23298" exitCode=60
Nov 02 13:15:18.295364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.295317    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk" event={"ID":"9262bfd3-6bbc-4161-afa6-93fd1a6639a1","Type":"ContainerDied","Data":"d710cb2ce57053310c79e8dc8b51bfc2b953fc97096897e9da9b760c45e23298"}
Nov 02 13:15:18.295736 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.295380    5879 scope.go:122] "RemoveContainer" containerID="d118e345741850bbfdd3f629b4b3b2ccc8aa74a00c0d78fcf06201a5899b4aef"
Nov 02 13:15:18.295834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.295816    5879 scope.go:122] "RemoveContainer" containerID="d710cb2ce57053310c79e8dc8b51bfc2b953fc97096897e9da9b760c45e23298"
Nov 02 13:15:18.296275 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:18.296248    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk_odh-ai-gateway-infra(9262bfd3-6bbc-4161-afa6-93fd1a6639a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1"
Nov 02 13:15:18.341051 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.341022    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"]
Nov 02 13:15:18.632199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.632168    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"
Nov 02 13:15:18.786644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.786612    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-maas-api-service-ca\") pod \"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05\" (UID: \"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05\") "
Nov 02 13:15:18.786872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.786734    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-kube-api-access-tmkqc\" (UniqueName: \"kubernetes.io/projected/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-kube-api-access-tmkqc\") pod \"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05\" (UID: \"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05\") "
Nov 02 13:15:18.787039 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.786980    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-maas-api-service-ca" pod "1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" (UID: "1e1b6178-a8b0-4b37-ae61-e2ce108e7a05"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:18.787660 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.787636    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:18.789184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.789158    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-kube-api-access-tmkqc" pod "1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" (UID: "1e1b6178-a8b0-4b37-ae61-e2ce108e7a05"). InnerVolumeSpecName "kube-api-access-tmkqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:18.889172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:18.889141    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmkqc\" (UniqueName: \"kubernetes.io/projected/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05-kube-api-access-tmkqc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:19.010422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.010388    5879 scope.go:122] "RemoveContainer" containerID="a5cea5ddb71df623f6b5031b6e3ff2d340a33a8b68c344b39a8c8a09f4d9c5e2"
Nov 02 13:15:19.011475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.010589    5879 scope.go:122] "RemoveContainer" containerID="d5d0d54313768f763250b8ee0be59416626ae75ab10e88f01f2a41187c9404da"
Nov 02 13:15:19.011475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.010685    5879 scope.go:122] "RemoveContainer" containerID="2c98a988d37251c0732ad78b772301f400f6bc777f7f93f0e32b743461df4f3b"
Nov 02 13:15:19.011475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.010758    5879 scope.go:122] "RemoveContainer" containerID="7783195e1d7939582645e7daf634638826e2aa5d7111433df53e937abd641c44"
Nov 02 13:15:19.311805 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.311776    5879 generic.go:356] "Generic (PLEG): container finished" podID="4c723d82-c700-4def-b8c0-081c5e358b18" containerID="f37d3fe5b2577db4dbc0bf02599b6907d0e50754d56080a24ebcf68c168e080f" exitCode=60
Nov 02 13:15:19.312005 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.311851    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl" event={"ID":"4c723d82-c700-4def-b8c0-081c5e358b18","Type":"ContainerDied","Data":"f37d3fe5b2577db4dbc0bf02599b6907d0e50754d56080a24ebcf68c168e080f"}
Nov 02 13:15:19.312005 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.311918    5879 scope.go:122] "RemoveContainer" containerID="5de7a9251738ba2a143a64c1d6b95b865c4d8a2824b6b4fa12ef3174b221702e"
Nov 02 13:15:19.312408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.312388    5879 scope.go:122] "RemoveContainer" containerID="f37d3fe5b2577db4dbc0bf02599b6907d0e50754d56080a24ebcf68c168e080f"
Nov 02 13:15:19.312872 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:19.312843    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl_odh-ai-gateway-infra(4c723d82-c700-4def-b8c0-081c5e358b18)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl" podUID="4c723d82-c700-4def-b8c0-081c5e358b18"
Nov 02 13:15:19.316571 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.316547    5879 generic.go:356] "Generic (PLEG): container finished" podID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" containerID="37cf015570b1ba34935fcb664e53d6dbb753de9380feb028905355a78485e269" exitCode=60
Nov 02 13:15:19.316688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.316612    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt" event={"ID":"5ffec64c-ff1f-42df-83a2-3d45751f8bb7","Type":"ContainerDied","Data":"37cf015570b1ba34935fcb664e53d6dbb753de9380feb028905355a78485e269"}
Nov 02 13:15:19.317152 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.317132    5879 scope.go:122] "RemoveContainer" containerID="37cf015570b1ba34935fcb664e53d6dbb753de9380feb028905355a78485e269"
Nov 02 13:15:19.317565 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:19.317535    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt_odh-ai-gateway-infra(5ffec64c-ff1f-42df-83a2-3d45751f8bb7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7"
Nov 02 13:15:19.319390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.319371    5879 generic.go:356] "Generic (PLEG): container finished" podID="e518dcfa-edd5-4c9c-b128-3504862ab477" containerID="a3a50bac38b3c670b3cf18e362c7786c58bbaaceb8360a5d6c5fbbe3aced61f2" exitCode=60
Nov 02 13:15:19.319511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.319450    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq" event={"ID":"e518dcfa-edd5-4c9c-b128-3504862ab477","Type":"ContainerDied","Data":"a3a50bac38b3c670b3cf18e362c7786c58bbaaceb8360a5d6c5fbbe3aced61f2"}
Nov 02 13:15:19.319871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.319854    5879 scope.go:122] "RemoveContainer" containerID="a3a50bac38b3c670b3cf18e362c7786c58bbaaceb8360a5d6c5fbbe3aced61f2"
Nov 02 13:15:19.320305 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:19.320252    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq_odh-ai-gateway-infra(e518dcfa-edd5-4c9c-b128-3504862ab477)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477"
Nov 02 13:15:19.321390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.321365    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"
Nov 02 13:15:19.321467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.321404    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78" event={"ID":"1e1b6178-a8b0-4b37-ae61-e2ce108e7a05","Type":"ContainerDied","Data":"03baab5ade87c4e823c19e42848001904cc644726519b04576a6a2a7ad720f39"}
Nov 02 13:15:19.366307 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.366273    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"]
Nov 02 13:15:19.481925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.481900    5879 scope.go:122] "RemoveContainer" containerID="8d887314fa0303726d5575e93d25ae097080e77e8d1a9d7e0a99197a2482c432"
Nov 02 13:15:19.541935 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.541902    5879 scope.go:122] "RemoveContainer" containerID="ce2e879f6d790f77b9b332de995435fbf9fe8f4ec1ebfca455734e8927e0aae1"
Nov 02 13:15:19.586471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.586445    5879 scope.go:122] "RemoveContainer" containerID="307e5d98c56ce476947cabf5dc1877a35dc4b52084516d909b5a147ee7cb5fad"
Nov 02 13:15:19.604718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.604670    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"]
Nov 02 13:15:19.614012 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.613968    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-14fcb6-29893755-vmm78"]
Nov 02 13:15:19.706835 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.706813    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"
Nov 02 13:15:19.724434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.724402    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-kube-api-access-8m7zb\" (UniqueName: \"kubernetes.io/projected/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-kube-api-access-8m7zb\") pod \"9262bfd3-6bbc-4161-afa6-93fd1a6639a1\" (UID: \"9262bfd3-6bbc-4161-afa6-93fd1a6639a1\") "
Nov 02 13:15:19.724614 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.724502    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-maas-api-service-ca\") pod \"9262bfd3-6bbc-4161-afa6-93fd1a6639a1\" (UID: \"9262bfd3-6bbc-4161-afa6-93fd1a6639a1\") "
Nov 02 13:15:19.724945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.724913    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-maas-api-service-ca" pod "9262bfd3-6bbc-4161-afa6-93fd1a6639a1" (UID: "9262bfd3-6bbc-4161-afa6-93fd1a6639a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:19.725790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.725759    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:19.727164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.727141    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-kube-api-access-8m7zb" pod "9262bfd3-6bbc-4161-afa6-93fd1a6639a1" (UID: "9262bfd3-6bbc-4161-afa6-93fd1a6639a1"). InnerVolumeSpecName "kube-api-access-8m7zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:19.827831 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.827787    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m7zb\" (UniqueName: \"kubernetes.io/projected/9262bfd3-6bbc-4161-afa6-93fd1a6639a1-kube-api-access-8m7zb\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:19.948752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:19.948722    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:20.010690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.010658    5879 scope.go:122] "RemoveContainer" containerID="f6239f5ca272786738d1a3dc16ea806998f2ed8036e98833c0558cd23ff40579"
Nov 02 13:15:20.011467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.011415    5879 scope.go:122] "RemoveContainer" containerID="f9432cf86284898c32e0cf65d24e012d7b215cf37a6ffeff9f76a9683f6c3d4e"
Nov 02 13:15:20.024975 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.024939    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" path="/var/lib/kubelet/pods/1e1b6178-a8b0-4b37-ae61-e2ce108e7a05/volumes"
Nov 02 13:15:20.333427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.333391    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd" event={"ID":"397a2f80-d67d-43d4-a4f8-35f6e93e111e","Type":"ContainerStarted","Data":"eac7f72c14976c037e7b9bbaa62d70c0662b1184a4029cd89f79147612097a9e"}
Nov 02 13:15:20.335596 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.335566    5879 generic.go:356] "Generic (PLEG): container finished" podID="265f8052-98a4-44a8-95ef-239d37e81c43" containerID="df79728333a288729cd42a79231948687fd9528234174673f1f306dcc6344c9b" exitCode=60
Nov 02 13:15:20.335770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.335644    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr" event={"ID":"265f8052-98a4-44a8-95ef-239d37e81c43","Type":"ContainerDied","Data":"df79728333a288729cd42a79231948687fd9528234174673f1f306dcc6344c9b"}
Nov 02 13:15:20.335770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.335699    5879 scope.go:122] "RemoveContainer" containerID="2c98a988d37251c0732ad78b772301f400f6bc777f7f93f0e32b743461df4f3b"
Nov 02 13:15:20.336189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.336171    5879 scope.go:122] "RemoveContainer" containerID="df79728333a288729cd42a79231948687fd9528234174673f1f306dcc6344c9b"
Nov 02 13:15:20.336623 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:20.336597    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr_odh-ai-gateway-infra(265f8052-98a4-44a8-95ef-239d37e81c43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr" podUID="265f8052-98a4-44a8-95ef-239d37e81c43"
Nov 02 13:15:20.338058 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.338032    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8" event={"ID":"c34bf9c9-3f3e-47d2-adaa-8f473dec78be","Type":"ContainerStarted","Data":"e60f70718335b85a0e6321528fba1726db3124a931285b832b1085ad1d38d5c8"}
Nov 02 13:15:20.339957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.339931    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5" event={"ID":"23754db5-04c9-48a5-841a-433830943be2","Type":"ContainerStarted","Data":"a38d68f44a4603fc6995eb163de71c6cc4a48bf356bde3f0c3efb193e2610ed0"}
Nov 02 13:15:20.343350 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.343322    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"
Nov 02 13:15:20.343461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.343342    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk" event={"ID":"9262bfd3-6bbc-4161-afa6-93fd1a6639a1","Type":"ContainerDied","Data":"7eb3299a0708771337374e08d2f77cb5ffa394a418b34087fb81901287ca58a5"}
Nov 02 13:15:20.353744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.353728    5879 scope.go:122] "RemoveContainer" containerID="d710cb2ce57053310c79e8dc8b51bfc2b953fc97096897e9da9b760c45e23298"
Nov 02 13:15:20.369944 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.369919    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"]
Nov 02 13:15:20.444060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.442856    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"]
Nov 02 13:15:20.484963 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.484862    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"]
Nov 02 13:15:20.522754 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.522515    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"]
Nov 02 13:15:20.535282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.535254    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893755-jthvk"]
Nov 02 13:15:20.697665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.697640    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"
Nov 02 13:15:20.744185 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.744083    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c723d82-c700-4def-b8c0-081c5e358b18-kube-api-access-5dgl7\" (UniqueName: \"kubernetes.io/projected/4c723d82-c700-4def-b8c0-081c5e358b18-kube-api-access-5dgl7\") pod \"4c723d82-c700-4def-b8c0-081c5e358b18\" (UID: \"4c723d82-c700-4def-b8c0-081c5e358b18\") "
Nov 02 13:15:20.744582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.744532    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c723d82-c700-4def-b8c0-081c5e358b18-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c723d82-c700-4def-b8c0-081c5e358b18-maas-api-service-ca\") pod \"4c723d82-c700-4def-b8c0-081c5e358b18\" (UID: \"4c723d82-c700-4def-b8c0-081c5e358b18\") "
Nov 02 13:15:20.745960 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.745916    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c723d82-c700-4def-b8c0-081c5e358b18-maas-api-service-ca" pod "4c723d82-c700-4def-b8c0-081c5e358b18" (UID: "4c723d82-c700-4def-b8c0-081c5e358b18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:20.746199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.746177    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c723d82-c700-4def-b8c0-081c5e358b18-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:20.752652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.752607    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c723d82-c700-4def-b8c0-081c5e358b18-kube-api-access-5dgl7" pod "4c723d82-c700-4def-b8c0-081c5e358b18" (UID: "4c723d82-c700-4def-b8c0-081c5e358b18"). InnerVolumeSpecName "kube-api-access-5dgl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:20.804815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.804679    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"
Nov 02 13:15:20.816760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.816733    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"
Nov 02 13:15:20.847018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.846961    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-maas-api-service-ca\") pod \"5ffec64c-ff1f-42df-83a2-3d45751f8bb7\" (UID: \"5ffec64c-ff1f-42df-83a2-3d45751f8bb7\") "
Nov 02 13:15:20.847018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.847014    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e518dcfa-edd5-4c9c-b128-3504862ab477-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e518dcfa-edd5-4c9c-b128-3504862ab477-maas-api-service-ca\") pod \"e518dcfa-edd5-4c9c-b128-3504862ab477\" (UID: \"e518dcfa-edd5-4c9c-b128-3504862ab477\") "
Nov 02 13:15:20.847231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.847052    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-kube-api-access-x66cp\" (UniqueName: \"kubernetes.io/projected/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-kube-api-access-x66cp\") pod \"5ffec64c-ff1f-42df-83a2-3d45751f8bb7\" (UID: \"5ffec64c-ff1f-42df-83a2-3d45751f8bb7\") "
Nov 02 13:15:20.847231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.847159    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e518dcfa-edd5-4c9c-b128-3504862ab477-kube-api-access-b4tc8\" (UniqueName: \"kubernetes.io/projected/e518dcfa-edd5-4c9c-b128-3504862ab477-kube-api-access-b4tc8\") pod \"e518dcfa-edd5-4c9c-b128-3504862ab477\" (UID: \"e518dcfa-edd5-4c9c-b128-3504862ab477\") "
Nov 02 13:15:20.847381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.847356    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-maas-api-service-ca" pod "5ffec64c-ff1f-42df-83a2-3d45751f8bb7" (UID: "5ffec64c-ff1f-42df-83a2-3d45751f8bb7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:20.847438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.847416    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e518dcfa-edd5-4c9c-b128-3504862ab477-maas-api-service-ca" pod "e518dcfa-edd5-4c9c-b128-3504862ab477" (UID: "e518dcfa-edd5-4c9c-b128-3504862ab477"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:20.847892 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.847873    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dgl7\" (UniqueName: \"kubernetes.io/projected/4c723d82-c700-4def-b8c0-081c5e358b18-kube-api-access-5dgl7\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:20.848090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.847896    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:20.848090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.847910    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e518dcfa-edd5-4c9c-b128-3504862ab477-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:20.851578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.851550    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e518dcfa-edd5-4c9c-b128-3504862ab477-kube-api-access-b4tc8" pod "e518dcfa-edd5-4c9c-b128-3504862ab477" (UID: "e518dcfa-edd5-4c9c-b128-3504862ab477"). InnerVolumeSpecName "kube-api-access-b4tc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:20.851746 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.851591    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-kube-api-access-x66cp" pod "5ffec64c-ff1f-42df-83a2-3d45751f8bb7" (UID: "5ffec64c-ff1f-42df-83a2-3d45751f8bb7"). InnerVolumeSpecName "kube-api-access-x66cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:20.949247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.949217    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x66cp\" (UniqueName: \"kubernetes.io/projected/5ffec64c-ff1f-42df-83a2-3d45751f8bb7-kube-api-access-x66cp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:20.949247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:20.949244    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4tc8\" (UniqueName: \"kubernetes.io/projected/e518dcfa-edd5-4c9c-b128-3504862ab477-kube-api-access-b4tc8\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:21.010269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.010235    5879 scope.go:122] "RemoveContainer" containerID="e608a17961c1da22e6ceedc10051cc61bb1b41496177cf4dbdd7ed25d18c9709"
Nov 02 13:15:21.010473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.010370    5879 scope.go:122] "RemoveContainer" containerID="8403eab313e8742ac0d0737a691aaf555c192b2c4bd8ff7aa66f3882a08a18ae"
Nov 02 13:15:21.010563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.010543    5879 scope.go:122] "RemoveContainer" containerID="c128541ac14a5f8ea7083fc905f45f761544966546093c827b4060d086111960"
Nov 02 13:15:21.010697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.010676    5879 scope.go:122] "RemoveContainer" containerID="0b743614ff0627827cc4143ee4ff03e456e1cf1d4f4b3cce6beb33e9975964e9"
Nov 02 13:15:21.010905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.010881    5879 scope.go:122] "RemoveContainer" containerID="00bece299034862b1cd2fedca7541f12495be204d8c73d6bfd2c06ac63e4565b"
Nov 02 13:15:21.360026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.359977    5879 generic.go:356] "Generic (PLEG): container finished" podID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" containerID="e60f70718335b85a0e6321528fba1726db3124a931285b832b1085ad1d38d5c8" exitCode=60
Nov 02 13:15:21.360026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.360024    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8" event={"ID":"c34bf9c9-3f3e-47d2-adaa-8f473dec78be","Type":"ContainerDied","Data":"e60f70718335b85a0e6321528fba1726db3124a931285b832b1085ad1d38d5c8"}
Nov 02 13:15:21.360320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.360088    5879 scope.go:122] "RemoveContainer" containerID="d5d0d54313768f763250b8ee0be59416626ae75ab10e88f01f2a41187c9404da"
Nov 02 13:15:21.360509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.360490    5879 scope.go:122] "RemoveContainer" containerID="e60f70718335b85a0e6321528fba1726db3124a931285b832b1085ad1d38d5c8"
Nov 02 13:15:21.360902 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:21.360874    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8_odh-ai-gateway-infra(c34bf9c9-3f3e-47d2-adaa-8f473dec78be)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be"
Nov 02 13:15:21.362436 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.362415    5879 generic.go:356] "Generic (PLEG): container finished" podID="23754db5-04c9-48a5-841a-433830943be2" containerID="a38d68f44a4603fc6995eb163de71c6cc4a48bf356bde3f0c3efb193e2610ed0" exitCode=60
Nov 02 13:15:21.362535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.362471    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5" event={"ID":"23754db5-04c9-48a5-841a-433830943be2","Type":"ContainerDied","Data":"a38d68f44a4603fc6995eb163de71c6cc4a48bf356bde3f0c3efb193e2610ed0"}
Nov 02 13:15:21.362892 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.362872    5879 scope.go:122] "RemoveContainer" containerID="a38d68f44a4603fc6995eb163de71c6cc4a48bf356bde3f0c3efb193e2610ed0"
Nov 02 13:15:21.363253 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:21.363223    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5_odh-ai-gateway-infra(23754db5-04c9-48a5-841a-433830943be2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5" podUID="23754db5-04c9-48a5-841a-433830943be2"
Nov 02 13:15:21.364095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.364081    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"
Nov 02 13:15:21.364095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.364107    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl" event={"ID":"4c723d82-c700-4def-b8c0-081c5e358b18","Type":"ContainerDied","Data":"fad7a6acebad6aa8811f2d71c71e4e2ece2e76fb5d3ae438a6c1f64bf4a108ae"}
Nov 02 13:15:21.367101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.367084    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"
Nov 02 13:15:21.367204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.367102    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt" event={"ID":"5ffec64c-ff1f-42df-83a2-3d45751f8bb7","Type":"ContainerDied","Data":"138e558331c36c3d42d593a7b29d02ab7320b740fba685255ffd6d7c439742c7"}
Nov 02 13:15:21.368735 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.368718    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"
Nov 02 13:15:21.368845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.368768    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq" event={"ID":"e518dcfa-edd5-4c9c-b128-3504862ab477","Type":"ContainerDied","Data":"b44e823ba3f432cdd02f24f294af723f895a57b1491c45cad8755490aec12612"}
Nov 02 13:15:21.371294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.371255    5879 generic.go:356] "Generic (PLEG): container finished" podID="2ba4d701-2d78-40cf-bacc-8206cc16e923" containerID="6d96aef23d2f0b93a62d61d225f910628f3f62c120f404ad780e18b1037f0678" exitCode=60
Nov 02 13:15:21.371387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.371311    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz" event={"ID":"2ba4d701-2d78-40cf-bacc-8206cc16e923","Type":"ContainerDied","Data":"6d96aef23d2f0b93a62d61d225f910628f3f62c120f404ad780e18b1037f0678"}
Nov 02 13:15:21.371747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.371729    5879 scope.go:122] "RemoveContainer" containerID="6d96aef23d2f0b93a62d61d225f910628f3f62c120f404ad780e18b1037f0678"
Nov 02 13:15:21.372166 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:21.372122    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz_odh-ai-gateway-infra(2ba4d701-2d78-40cf-bacc-8206cc16e923)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923"
Nov 02 13:15:21.374057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.374035    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9" event={"ID":"021f486a-e073-4944-8b61-ecece24ff7fe","Type":"ContainerDied","Data":"3f8f2f9be75ca39cca55657c123978c1f15529244f88429410c82be32b592914"}
Nov 02 13:15:21.374175 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.373987    5879 generic.go:356] "Generic (PLEG): container finished" podID="021f486a-e073-4944-8b61-ecece24ff7fe" containerID="3f8f2f9be75ca39cca55657c123978c1f15529244f88429410c82be32b592914" exitCode=60
Nov 02 13:15:21.374395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.374375    5879 scope.go:122] "RemoveContainer" containerID="3f8f2f9be75ca39cca55657c123978c1f15529244f88429410c82be32b592914"
Nov 02 13:15:21.374666 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:21.374647    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9_odh-ai-gateway-infra(021f486a-e073-4944-8b61-ecece24ff7fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9" podUID="021f486a-e073-4944-8b61-ecece24ff7fe"
Nov 02 13:15:21.377125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.377097    5879 generic.go:356] "Generic (PLEG): container finished" podID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" containerID="eac7f72c14976c037e7b9bbaa62d70c0662b1184a4029cd89f79147612097a9e" exitCode=60
Nov 02 13:15:21.377235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.377220    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd" event={"ID":"397a2f80-d67d-43d4-a4f8-35f6e93e111e","Type":"ContainerDied","Data":"eac7f72c14976c037e7b9bbaa62d70c0662b1184a4029cd89f79147612097a9e"}
Nov 02 13:15:21.377610 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.377584    5879 scope.go:122] "RemoveContainer" containerID="eac7f72c14976c037e7b9bbaa62d70c0662b1184a4029cd89f79147612097a9e"
Nov 02 13:15:21.377845 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:21.377828    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd_odh-ai-gateway-infra(397a2f80-d67d-43d4-a4f8-35f6e93e111e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e"
Nov 02 13:15:21.395074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.395046    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"]
Nov 02 13:15:21.430957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.430934    5879 scope.go:122] "RemoveContainer" containerID="7783195e1d7939582645e7daf634638826e2aa5d7111433df53e937abd641c44"
Nov 02 13:15:21.440552 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.440529    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"]
Nov 02 13:15:21.449888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.449866    5879 scope.go:122] "RemoveContainer" containerID="f37d3fe5b2577db4dbc0bf02599b6907d0e50754d56080a24ebcf68c168e080f"
Nov 02 13:15:21.467903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.467878    5879 scope.go:122] "RemoveContainer" containerID="37cf015570b1ba34935fcb664e53d6dbb753de9380feb028905355a78485e269"
Nov 02 13:15:21.480406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.480385    5879 scope.go:122] "RemoveContainer" containerID="a3a50bac38b3c670b3cf18e362c7786c58bbaaceb8360a5d6c5fbbe3aced61f2"
Nov 02 13:15:21.490627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.490598    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"]
Nov 02 13:15:21.504108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.504085    5879 scope.go:122] "RemoveContainer" containerID="f6239f5ca272786738d1a3dc16ea806998f2ed8036e98833c0558cd23ff40579"
Nov 02 13:15:21.524689 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.524667    5879 scope.go:122] "RemoveContainer" containerID="f9432cf86284898c32e0cf65d24e012d7b215cf37a6ffeff9f76a9683f6c3d4e"
Nov 02 13:15:21.541606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.541581    5879 scope.go:122] "RemoveContainer" containerID="a5cea5ddb71df623f6b5031b6e3ff2d340a33a8b68c344b39a8c8a09f4d9c5e2"
Nov 02 13:15:21.634615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.634579    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"]
Nov 02 13:15:21.683826 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.683790    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"]
Nov 02 13:15:21.691522 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.691486    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893755-xl5zq"]
Nov 02 13:15:21.725257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.725073    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"]
Nov 02 13:15:21.732289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.732240    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0e422b-29893755-v7vsl"]
Nov 02 13:15:21.806395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.806355    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"]
Nov 02 13:15:21.813313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.813278    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-068756-29893755-v4pvt"]
Nov 02 13:15:21.846411 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.846386    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"
Nov 02 13:15:21.847502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.847477    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:21.871083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.866438    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/265f8052-98a4-44a8-95ef-239d37e81c43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/265f8052-98a4-44a8-95ef-239d37e81c43-maas-api-service-ca\") pod \"265f8052-98a4-44a8-95ef-239d37e81c43\" (UID: \"265f8052-98a4-44a8-95ef-239d37e81c43\") "
Nov 02 13:15:21.871083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.866803    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265f8052-98a4-44a8-95ef-239d37e81c43-maas-api-service-ca" pod "265f8052-98a4-44a8-95ef-239d37e81c43" (UID: "265f8052-98a4-44a8-95ef-239d37e81c43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:21.871083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.866974    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265f8052-98a4-44a8-95ef-239d37e81c43-kube-api-access-hsnxl\" (UniqueName: \"kubernetes.io/projected/265f8052-98a4-44a8-95ef-239d37e81c43-kube-api-access-hsnxl\") pod \"265f8052-98a4-44a8-95ef-239d37e81c43\" (UID: \"265f8052-98a4-44a8-95ef-239d37e81c43\") "
Nov 02 13:15:21.871083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.868552    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/265f8052-98a4-44a8-95ef-239d37e81c43-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:21.913399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.913353    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265f8052-98a4-44a8-95ef-239d37e81c43-kube-api-access-hsnxl" pod "265f8052-98a4-44a8-95ef-239d37e81c43" (UID: "265f8052-98a4-44a8-95ef-239d37e81c43"). InnerVolumeSpecName "kube-api-access-hsnxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:21.970593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:21.970554    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsnxl\" (UniqueName: \"kubernetes.io/projected/265f8052-98a4-44a8-95ef-239d37e81c43-kube-api-access-hsnxl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:22.010164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.010133    5879 scope.go:122] "RemoveContainer" containerID="fe5670dcd2998a923d073f1e7ce62be5a61544ae034fb7e6280627541067182d"
Nov 02 13:15:22.010310 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.010229    5879 scope.go:122] "RemoveContainer" containerID="9bd72b084452751467823ba6e1f1cf8f3ff540777714df46d2306e6a708a03be"
Nov 02 13:15:22.021913 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.021884    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c723d82-c700-4def-b8c0-081c5e358b18" path="/var/lib/kubelet/pods/4c723d82-c700-4def-b8c0-081c5e358b18/volumes"
Nov 02 13:15:22.022418 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.022401    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" path="/var/lib/kubelet/pods/5ffec64c-ff1f-42df-83a2-3d45751f8bb7/volumes"
Nov 02 13:15:22.022829 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.022812    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" path="/var/lib/kubelet/pods/9262bfd3-6bbc-4161-afa6-93fd1a6639a1/volumes"
Nov 02 13:15:22.023586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.023565    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477" path="/var/lib/kubelet/pods/e518dcfa-edd5-4c9c-b128-3504862ab477/volumes"
Nov 02 13:15:22.396148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.396124    5879 generic.go:356] "Generic (PLEG): container finished" podID="4863e18b-0942-4e01-94e1-293fd89aa56f" containerID="daf0947be532ecf57dedc1f21b30c90e95f5507341cb208dbc904dba039cc924" exitCode=60
Nov 02 13:15:22.396275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.396194    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw" event={"ID":"4863e18b-0942-4e01-94e1-293fd89aa56f","Type":"ContainerDied","Data":"daf0947be532ecf57dedc1f21b30c90e95f5507341cb208dbc904dba039cc924"}
Nov 02 13:15:22.396275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.396261    5879 scope.go:122] "RemoveContainer" containerID="00bece299034862b1cd2fedca7541f12495be204d8c73d6bfd2c06ac63e4565b"
Nov 02 13:15:22.396775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.396756    5879 scope.go:122] "RemoveContainer" containerID="daf0947be532ecf57dedc1f21b30c90e95f5507341cb208dbc904dba039cc924"
Nov 02 13:15:22.397337 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:22.397253    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw_odh-ai-gateway-infra(4863e18b-0942-4e01-94e1-293fd89aa56f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f"
Nov 02 13:15:22.401658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.401637    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" event={"ID":"d8964b32-2de3-4bb6-b25c-7d7a561d0251","Type":"ContainerStarted","Data":"4cbdc141c97883c4ed10f25a43e24b49935173a78df80f87e3e51af2c737d2e6"}
Nov 02 13:15:22.402129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.402110    5879 scope.go:122] "RemoveContainer" containerID="4cbdc141c97883c4ed10f25a43e24b49935173a78df80f87e3e51af2c737d2e6"
Nov 02 13:15:22.403033 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:22.402464    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c_odh-ai-gateway-infra(d8964b32-2de3-4bb6-b25c-7d7a561d0251)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251"
Nov 02 13:15:22.405911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.405889    5879 generic.go:356] "Generic (PLEG): container finished" podID="20982710-74e3-4acb-af83-0f1d76338674" containerID="76903ad7de711f054d8b900cce2adb2f16ffc0264520dc32f94cdc2cb5e6d0b9" exitCode=60
Nov 02 13:15:22.406044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.405957    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg" event={"ID":"20982710-74e3-4acb-af83-0f1d76338674","Type":"ContainerDied","Data":"76903ad7de711f054d8b900cce2adb2f16ffc0264520dc32f94cdc2cb5e6d0b9"}
Nov 02 13:15:22.406403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.406383    5879 scope.go:122] "RemoveContainer" containerID="76903ad7de711f054d8b900cce2adb2f16ffc0264520dc32f94cdc2cb5e6d0b9"
Nov 02 13:15:22.406689 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:22.406667    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg_odh-ai-gateway-infra(20982710-74e3-4acb-af83-0f1d76338674)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg" podUID="20982710-74e3-4acb-af83-0f1d76338674"
Nov 02 13:15:22.408048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.408025    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt" event={"ID":"9025c2c7-57b9-439c-bbf8-91a5fd1433c0","Type":"ContainerStarted","Data":"fbe8b466c54dc29ffb9fa3ddc292d73252b9db6008b652f84e4243f50d068081"}
Nov 02 13:15:22.409985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.409965    5879 generic.go:356] "Generic (PLEG): container finished" podID="e9110a44-4129-4c65-9de1-f2574890c394" containerID="bca019f2c127c76aab7af8c2770affa7065b428337ef07af35fd434cf3db6d12" exitCode=60
Nov 02 13:15:22.410090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.410012    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv" event={"ID":"e9110a44-4129-4c65-9de1-f2574890c394","Type":"ContainerDied","Data":"bca019f2c127c76aab7af8c2770affa7065b428337ef07af35fd434cf3db6d12"}
Nov 02 13:15:22.410389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.410371    5879 scope.go:122] "RemoveContainer" containerID="bca019f2c127c76aab7af8c2770affa7065b428337ef07af35fd434cf3db6d12"
Nov 02 13:15:22.410647 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:22.410629    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv_odh-ai-gateway-infra(e9110a44-4129-4c65-9de1-f2574890c394)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv" podUID="e9110a44-4129-4c65-9de1-f2574890c394"
Nov 02 13:15:22.415298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.415279    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr" event={"ID":"265f8052-98a4-44a8-95ef-239d37e81c43","Type":"ContainerDied","Data":"892ec15c296f58c0de372c93dcc0862f0a92b98fd2b7fe941da76586e2bcd3d1"}
Nov 02 13:15:22.415298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.415291    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"
Nov 02 13:15:22.471456 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.471387    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"]
Nov 02 13:15:22.521974 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.518104    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"]
Nov 02 13:15:22.631151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.630770    5879 scope.go:122] "RemoveContainer" containerID="e608a17961c1da22e6ceedc10051cc61bb1b41496177cf4dbdd7ed25d18c9709"
Nov 02 13:15:22.643608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.643570    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"
Nov 02 13:15:22.663915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.663886    5879 scope.go:122] "RemoveContainer" containerID="c128541ac14a5f8ea7083fc905f45f761544966546093c827b4060d086111960"
Nov 02 13:15:22.666367 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:22.666325    5879 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9025c2c7_57b9_439c_bbf8_91a5fd1433c0.slice/crio-fbe8b466c54dc29ffb9fa3ddc292d73252b9db6008b652f84e4243f50d068081.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9025c2c7_57b9_439c_bbf8_91a5fd1433c0.slice/crio-conmon-fbe8b466c54dc29ffb9fa3ddc292d73252b9db6008b652f84e4243f50d068081.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 13:15:22.682624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.682582    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-maas-api-service-ca\") pod \"c34bf9c9-3f3e-47d2-adaa-8f473dec78be\" (UID: \"c34bf9c9-3f3e-47d2-adaa-8f473dec78be\") "
Nov 02 13:15:22.683253 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.683230    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-kube-api-access-cvgj6\" (UniqueName: \"kubernetes.io/projected/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-kube-api-access-cvgj6\") pod \"c34bf9c9-3f3e-47d2-adaa-8f473dec78be\" (UID: \"c34bf9c9-3f3e-47d2-adaa-8f473dec78be\") "
Nov 02 13:15:22.684417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.683098    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-maas-api-service-ca" pod "c34bf9c9-3f3e-47d2-adaa-8f473dec78be" (UID: "c34bf9c9-3f3e-47d2-adaa-8f473dec78be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:22.687739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.687717    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:22.688135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.688114    5879 scope.go:122] "RemoveContainer" containerID="df79728333a288729cd42a79231948687fd9528234174673f1f306dcc6344c9b"
Nov 02 13:15:22.782520 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.781876    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-kube-api-access-cvgj6" pod "c34bf9c9-3f3e-47d2-adaa-8f473dec78be" (UID: "c34bf9c9-3f3e-47d2-adaa-8f473dec78be"). InnerVolumeSpecName "kube-api-access-cvgj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:22.789319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.789284    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvgj6\" (UniqueName: \"kubernetes.io/projected/c34bf9c9-3f3e-47d2-adaa-8f473dec78be-kube-api-access-cvgj6\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:22.796345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.796319    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"
Nov 02 13:15:22.801932 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.801907    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"
Nov 02 13:15:22.899702 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.899652    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23754db5-04c9-48a5-841a-433830943be2-kube-api-access-hw72z\" (UniqueName: \"kubernetes.io/projected/23754db5-04c9-48a5-841a-433830943be2-kube-api-access-hw72z\") pod \"23754db5-04c9-48a5-841a-433830943be2\" (UID: \"23754db5-04c9-48a5-841a-433830943be2\") "
Nov 02 13:15:22.899865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.899753    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23754db5-04c9-48a5-841a-433830943be2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23754db5-04c9-48a5-841a-433830943be2-maas-api-service-ca\") pod \"23754db5-04c9-48a5-841a-433830943be2\" (UID: \"23754db5-04c9-48a5-841a-433830943be2\") "
Nov 02 13:15:22.899928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.899896    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/397a2f80-d67d-43d4-a4f8-35f6e93e111e-kube-api-access-rpxzn\" (UniqueName: \"kubernetes.io/projected/397a2f80-d67d-43d4-a4f8-35f6e93e111e-kube-api-access-rpxzn\") pod \"397a2f80-d67d-43d4-a4f8-35f6e93e111e\" (UID: \"397a2f80-d67d-43d4-a4f8-35f6e93e111e\") "
Nov 02 13:15:22.900023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.899970    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/397a2f80-d67d-43d4-a4f8-35f6e93e111e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/397a2f80-d67d-43d4-a4f8-35f6e93e111e-maas-api-service-ca\") pod \"397a2f80-d67d-43d4-a4f8-35f6e93e111e\" (UID: \"397a2f80-d67d-43d4-a4f8-35f6e93e111e\") "
Nov 02 13:15:22.901342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.901311    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397a2f80-d67d-43d4-a4f8-35f6e93e111e-maas-api-service-ca" pod "397a2f80-d67d-43d4-a4f8-35f6e93e111e" (UID: "397a2f80-d67d-43d4-a4f8-35f6e93e111e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:22.905903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.905868    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23754db5-04c9-48a5-841a-433830943be2-maas-api-service-ca" pod "23754db5-04c9-48a5-841a-433830943be2" (UID: "23754db5-04c9-48a5-841a-433830943be2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:22.919205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.919175    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397a2f80-d67d-43d4-a4f8-35f6e93e111e-kube-api-access-rpxzn" pod "397a2f80-d67d-43d4-a4f8-35f6e93e111e" (UID: "397a2f80-d67d-43d4-a4f8-35f6e93e111e"). InnerVolumeSpecName "kube-api-access-rpxzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:22.922788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.922738    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23754db5-04c9-48a5-841a-433830943be2-kube-api-access-hw72z" pod "23754db5-04c9-48a5-841a-433830943be2" (UID: "23754db5-04c9-48a5-841a-433830943be2"). InnerVolumeSpecName "kube-api-access-hw72z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:22.928214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.928178    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"
Nov 02 13:15:22.942211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.942182    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"
Nov 02 13:15:22.955711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.954685    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"]
Nov 02 13:15:22.960474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:22.960455    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ef5efe-29893755-7v8cr"]
Nov 02 13:15:23.001786 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.001758    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba4d701-2d78-40cf-bacc-8206cc16e923-kube-api-access-m7dt7\" (UniqueName: \"kubernetes.io/projected/2ba4d701-2d78-40cf-bacc-8206cc16e923-kube-api-access-m7dt7\") pod \"2ba4d701-2d78-40cf-bacc-8206cc16e923\" (UID: \"2ba4d701-2d78-40cf-bacc-8206cc16e923\") "
Nov 02 13:15:23.001922 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.001856    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba4d701-2d78-40cf-bacc-8206cc16e923-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba4d701-2d78-40cf-bacc-8206cc16e923-maas-api-service-ca\") pod \"2ba4d701-2d78-40cf-bacc-8206cc16e923\" (UID: \"2ba4d701-2d78-40cf-bacc-8206cc16e923\") "
Nov 02 13:15:23.002021 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.001987    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/021f486a-e073-4944-8b61-ecece24ff7fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/021f486a-e073-4944-8b61-ecece24ff7fe-maas-api-service-ca\") pod \"021f486a-e073-4944-8b61-ecece24ff7fe\" (UID: \"021f486a-e073-4944-8b61-ecece24ff7fe\") "
Nov 02 13:15:23.002077 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.002039    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021f486a-e073-4944-8b61-ecece24ff7fe-kube-api-access-5ccqx\" (UniqueName: \"kubernetes.io/projected/021f486a-e073-4944-8b61-ecece24ff7fe-kube-api-access-5ccqx\") pod \"021f486a-e073-4944-8b61-ecece24ff7fe\" (UID: \"021f486a-e073-4944-8b61-ecece24ff7fe\") "
Nov 02 13:15:23.002322 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.002297    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba4d701-2d78-40cf-bacc-8206cc16e923-maas-api-service-ca" pod "2ba4d701-2d78-40cf-bacc-8206cc16e923" (UID: "2ba4d701-2d78-40cf-bacc-8206cc16e923"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:23.002397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.002352    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021f486a-e073-4944-8b61-ecece24ff7fe-maas-api-service-ca" pod "021f486a-e073-4944-8b61-ecece24ff7fe" (UID: "021f486a-e073-4944-8b61-ecece24ff7fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:23.002688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.002666    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpxzn\" (UniqueName: \"kubernetes.io/projected/397a2f80-d67d-43d4-a4f8-35f6e93e111e-kube-api-access-rpxzn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:23.002814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.002798    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ba4d701-2d78-40cf-bacc-8206cc16e923-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:23.002896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.002818    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/397a2f80-d67d-43d4-a4f8-35f6e93e111e-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:23.002896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.002832    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/021f486a-e073-4944-8b61-ecece24ff7fe-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:23.002896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.002846    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw72z\" (UniqueName: \"kubernetes.io/projected/23754db5-04c9-48a5-841a-433830943be2-kube-api-access-hw72z\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:23.002896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.002861    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23754db5-04c9-48a5-841a-433830943be2-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:23.081781 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.081670    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba4d701-2d78-40cf-bacc-8206cc16e923-kube-api-access-m7dt7" pod "2ba4d701-2d78-40cf-bacc-8206cc16e923" (UID: "2ba4d701-2d78-40cf-bacc-8206cc16e923"). InnerVolumeSpecName "kube-api-access-m7dt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:23.082246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.082220    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021f486a-e073-4944-8b61-ecece24ff7fe-kube-api-access-5ccqx" pod "021f486a-e073-4944-8b61-ecece24ff7fe" (UID: "021f486a-e073-4944-8b61-ecece24ff7fe"). InnerVolumeSpecName "kube-api-access-5ccqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:23.105472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.105446    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7dt7\" (UniqueName: \"kubernetes.io/projected/2ba4d701-2d78-40cf-bacc-8206cc16e923-kube-api-access-m7dt7\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:23.105472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.105475    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ccqx\" (UniqueName: \"kubernetes.io/projected/021f486a-e073-4944-8b61-ecece24ff7fe-kube-api-access-5ccqx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:23.431857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.431821    5879 generic.go:356] "Generic (PLEG): container finished" podID="e97847d4-325d-4ce4-874d-199279b16a87" containerID="75e8e7bf93ecc05ba1dc9ecf27fbd1891e5d14ae02787d29dc9b447c349b4770" exitCode=60
Nov 02 13:15:23.432071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.431903    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82" event={"ID":"e97847d4-325d-4ce4-874d-199279b16a87","Type":"ContainerDied","Data":"75e8e7bf93ecc05ba1dc9ecf27fbd1891e5d14ae02787d29dc9b447c349b4770"}
Nov 02 13:15:23.432071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.431978    5879 scope.go:122] "RemoveContainer" containerID="9bd72b084452751467823ba6e1f1cf8f3ff540777714df46d2306e6a708a03be"
Nov 02 13:15:23.432518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.432496    5879 scope.go:122] "RemoveContainer" containerID="75e8e7bf93ecc05ba1dc9ecf27fbd1891e5d14ae02787d29dc9b447c349b4770"
Nov 02 13:15:23.432927 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:23.432904    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82_odh-ai-gateway-infra(e97847d4-325d-4ce4-874d-199279b16a87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82" podUID="e97847d4-325d-4ce4-874d-199279b16a87"
Nov 02 13:15:23.436030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.436005    5879 generic.go:356] "Generic (PLEG): container finished" podID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" containerID="4cbdc141c97883c4ed10f25a43e24b49935173a78df80f87e3e51af2c737d2e6" exitCode=60
Nov 02 13:15:23.436149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.436029    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" event={"ID":"d8964b32-2de3-4bb6-b25c-7d7a561d0251","Type":"ContainerDied","Data":"4cbdc141c97883c4ed10f25a43e24b49935173a78df80f87e3e51af2c737d2e6"}
Nov 02 13:15:23.436507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.436484    5879 scope.go:122] "RemoveContainer" containerID="4cbdc141c97883c4ed10f25a43e24b49935173a78df80f87e3e51af2c737d2e6"
Nov 02 13:15:23.436914 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:23.436891    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c_odh-ai-gateway-infra(d8964b32-2de3-4bb6-b25c-7d7a561d0251)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251"
Nov 02 13:15:23.437691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.437674    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"
Nov 02 13:15:23.437691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.437679    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz" event={"ID":"2ba4d701-2d78-40cf-bacc-8206cc16e923","Type":"ContainerDied","Data":"b9c5993aa6de4846a104f83ae08a1e4e955eb4fed655aa3685e6d78fbed4ef4f"}
Nov 02 13:15:23.441803 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.441783    5879 generic.go:356] "Generic (PLEG): container finished" podID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" containerID="fbe8b466c54dc29ffb9fa3ddc292d73252b9db6008b652f84e4243f50d068081" exitCode=60
Nov 02 13:15:23.441889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.441861    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt" event={"ID":"9025c2c7-57b9-439c-bbf8-91a5fd1433c0","Type":"ContainerDied","Data":"fbe8b466c54dc29ffb9fa3ddc292d73252b9db6008b652f84e4243f50d068081"}
Nov 02 13:15:23.442324 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.442302    5879 scope.go:122] "RemoveContainer" containerID="fbe8b466c54dc29ffb9fa3ddc292d73252b9db6008b652f84e4243f50d068081"
Nov 02 13:15:23.442670 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:23.442650    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt_odh-ai-gateway-infra(9025c2c7-57b9-439c-bbf8-91a5fd1433c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0"
Nov 02 13:15:23.445345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.445325    5879 generic.go:356] "Generic (PLEG): container finished" podID="031e33ac-9ee4-4153-9bdb-5289b569a146" containerID="9b30cadde4342f4abf843fd13ae7e568c8f599ae5ac9b0f031abc69b2108ac6b" exitCode=60
Nov 02 13:15:23.445445 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.445394    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2" event={"ID":"031e33ac-9ee4-4153-9bdb-5289b569a146","Type":"ContainerDied","Data":"9b30cadde4342f4abf843fd13ae7e568c8f599ae5ac9b0f031abc69b2108ac6b"}
Nov 02 13:15:23.445790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.445775    5879 scope.go:122] "RemoveContainer" containerID="9b30cadde4342f4abf843fd13ae7e568c8f599ae5ac9b0f031abc69b2108ac6b"
Nov 02 13:15:23.446155 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:23.446132    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2_odh-ai-gateway-infra(031e33ac-9ee4-4153-9bdb-5289b569a146)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146"
Nov 02 13:15:23.446891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.446864    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"
Nov 02 13:15:23.447029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.446909    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9" event={"ID":"021f486a-e073-4944-8b61-ecece24ff7fe","Type":"ContainerDied","Data":"dac3eb7e9866f45d8a586c3296ab1dc2db00c5f768492a4b7defc76632fed732"}
Nov 02 13:15:23.448475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.448456    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"
Nov 02 13:15:23.448580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.448484    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd" event={"ID":"397a2f80-d67d-43d4-a4f8-35f6e93e111e","Type":"ContainerDied","Data":"6e386d0ae35b3f86ba53416a864471f27b2763314351c12b613980d8386456f9"}
Nov 02 13:15:23.450941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.450909    5879 scope.go:122] "RemoveContainer" containerID="8403eab313e8742ac0d0737a691aaf555c192b2c4bd8ff7aa66f3882a08a18ae"
Nov 02 13:15:23.453357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.453332    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8" event={"ID":"c34bf9c9-3f3e-47d2-adaa-8f473dec78be","Type":"ContainerDied","Data":"4883e762c7b309ab20ed049b5a7a1699444938adc49f9716b725606ff26c02bb"}
Nov 02 13:15:23.453448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.453419    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"
Nov 02 13:15:23.459294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.458487    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5" event={"ID":"23754db5-04c9-48a5-841a-433830943be2","Type":"ContainerDied","Data":"c7480895742266210a2bcbfed68be71eff661cb2f41f16793026abeabf1fad4e"}
Nov 02 13:15:23.459294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.458785    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"
Nov 02 13:15:23.460056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.460021    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"]
Nov 02 13:15:23.473923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.473877    5879 scope.go:122] "RemoveContainer" containerID="6d96aef23d2f0b93a62d61d225f910628f3f62c120f404ad780e18b1037f0678"
Nov 02 13:15:23.487854 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.487138    5879 scope.go:122] "RemoveContainer" containerID="0b743614ff0627827cc4143ee4ff03e456e1cf1d4f4b3cce6beb33e9975964e9"
Nov 02 13:15:23.527438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.527415    5879 scope.go:122] "RemoveContainer" containerID="fe5670dcd2998a923d073f1e7ce62be5a61544ae034fb7e6280627541067182d"
Nov 02 13:15:23.556094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.553231    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"]
Nov 02 13:15:23.572888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.572861    5879 scope.go:122] "RemoveContainer" containerID="3f8f2f9be75ca39cca55657c123978c1f15529244f88429410c82be32b592914"
Nov 02 13:15:23.600031 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.585145    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:23.632787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.632700    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a6f2c0-29893755-rk2nd"]
Nov 02 13:15:23.637462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.637432    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"]
Nov 02 13:15:23.638613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.638568    5879 scope.go:122] "RemoveContainer" containerID="eac7f72c14976c037e7b9bbaa62d70c0662b1184a4029cd89f79147612097a9e"
Nov 02 13:15:23.651284 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.651217    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"]
Nov 02 13:15:23.656284 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.656259    5879 scope.go:122] "RemoveContainer" containerID="e60f70718335b85a0e6321528fba1726db3124a931285b832b1085ad1d38d5c8"
Nov 02 13:15:23.691624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.691433    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"]
Nov 02 13:15:23.714185 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.714145    5879 scope.go:122] "RemoveContainer" containerID="a38d68f44a4603fc6995eb163de71c6cc4a48bf356bde3f0c3efb193e2610ed0"
Nov 02 13:15:23.719655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.719551    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"
Nov 02 13:15:23.761218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.761185    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"]
Nov 02 13:15:23.789494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.789456    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"]
Nov 02 13:15:23.794161 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.794120    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-740f09-29893755-hhkq9"]
Nov 02 13:15:23.823582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.823546    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4863e18b-0942-4e01-94e1-293fd89aa56f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4863e18b-0942-4e01-94e1-293fd89aa56f-maas-api-service-ca\") pod \"4863e18b-0942-4e01-94e1-293fd89aa56f\" (UID: \"4863e18b-0942-4e01-94e1-293fd89aa56f\") "
Nov 02 13:15:23.823893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.823871    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4863e18b-0942-4e01-94e1-293fd89aa56f-kube-api-access-s4msn\" (UniqueName: \"kubernetes.io/projected/4863e18b-0942-4e01-94e1-293fd89aa56f-kube-api-access-s4msn\") pod \"4863e18b-0942-4e01-94e1-293fd89aa56f\" (UID: \"4863e18b-0942-4e01-94e1-293fd89aa56f\") "
Nov 02 13:15:23.824413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.824072    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4863e18b-0942-4e01-94e1-293fd89aa56f-maas-api-service-ca" pod "4863e18b-0942-4e01-94e1-293fd89aa56f" (UID: "4863e18b-0942-4e01-94e1-293fd89aa56f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:23.824716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.824690    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4863e18b-0942-4e01-94e1-293fd89aa56f-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:23.826824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.826801    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4863e18b-0942-4e01-94e1-293fd89aa56f-kube-api-access-s4msn" pod "4863e18b-0942-4e01-94e1-293fd89aa56f" (UID: "4863e18b-0942-4e01-94e1-293fd89aa56f"). InnerVolumeSpecName "kube-api-access-s4msn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:23.838171 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.838139    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"
Nov 02 13:15:23.880797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.880755    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"]
Nov 02 13:15:23.886492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.886462    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893755-jdppz"]
Nov 02 13:15:23.918131 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.918107    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"
Nov 02 13:15:23.925142 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.925111    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"]
Nov 02 13:15:23.925619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.925598    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9110a44-4129-4c65-9de1-f2574890c394-kube-api-access-nfv62\" (UniqueName: \"kubernetes.io/projected/e9110a44-4129-4c65-9de1-f2574890c394-kube-api-access-nfv62\") pod \"e9110a44-4129-4c65-9de1-f2574890c394\" (UID: \"e9110a44-4129-4c65-9de1-f2574890c394\") "
Nov 02 13:15:23.925694 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.925635    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9110a44-4129-4c65-9de1-f2574890c394-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9110a44-4129-4c65-9de1-f2574890c394-maas-api-service-ca\") pod \"e9110a44-4129-4c65-9de1-f2574890c394\" (UID: \"e9110a44-4129-4c65-9de1-f2574890c394\") "
Nov 02 13:15:23.926090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.926063    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9110a44-4129-4c65-9de1-f2574890c394-maas-api-service-ca" pod "e9110a44-4129-4c65-9de1-f2574890c394" (UID: "e9110a44-4129-4c65-9de1-f2574890c394"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:23.926524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.926504    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9110a44-4129-4c65-9de1-f2574890c394-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:23.926609 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.926525    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4msn\" (UniqueName: \"kubernetes.io/projected/4863e18b-0942-4e01-94e1-293fd89aa56f-kube-api-access-s4msn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:23.928194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.928169    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9110a44-4129-4c65-9de1-f2574890c394-kube-api-access-nfv62" pod "e9110a44-4129-4c65-9de1-f2574890c394" (UID: "e9110a44-4129-4c65-9de1-f2574890c394"). InnerVolumeSpecName "kube-api-access-nfv62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:23.929859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.929839    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-936529-29893755-75dx5"]
Nov 02 13:15:23.964683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.964641    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"]
Nov 02 13:15:23.968670 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:23.968648    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893755-zscx8"]
Nov 02 13:15:24.025710 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.025675    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021f486a-e073-4944-8b61-ecece24ff7fe" path="/var/lib/kubelet/pods/021f486a-e073-4944-8b61-ecece24ff7fe/volumes"
Nov 02 13:15:24.026334 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.026311    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23754db5-04c9-48a5-841a-433830943be2" path="/var/lib/kubelet/pods/23754db5-04c9-48a5-841a-433830943be2/volumes"
Nov 02 13:15:24.026807 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.026786    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="265f8052-98a4-44a8-95ef-239d37e81c43" path="/var/lib/kubelet/pods/265f8052-98a4-44a8-95ef-239d37e81c43/volumes"
Nov 02 13:15:24.029162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.027660    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20982710-74e3-4acb-af83-0f1d76338674-kube-api-access-hcl8h\" (UniqueName: \"kubernetes.io/projected/20982710-74e3-4acb-af83-0f1d76338674-kube-api-access-hcl8h\") pod \"20982710-74e3-4acb-af83-0f1d76338674\" (UID: \"20982710-74e3-4acb-af83-0f1d76338674\") "
Nov 02 13:15:24.029162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.027725    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923" path="/var/lib/kubelet/pods/2ba4d701-2d78-40cf-bacc-8206cc16e923/volumes"
Nov 02 13:15:24.029162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.027904    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20982710-74e3-4acb-af83-0f1d76338674-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20982710-74e3-4acb-af83-0f1d76338674-maas-api-service-ca\") pod \"20982710-74e3-4acb-af83-0f1d76338674\" (UID: \"20982710-74e3-4acb-af83-0f1d76338674\") "
Nov 02 13:15:24.029162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.028283    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" path="/var/lib/kubelet/pods/397a2f80-d67d-43d4-a4f8-35f6e93e111e/volumes"
Nov 02 13:15:24.029480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.029171    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" path="/var/lib/kubelet/pods/c34bf9c9-3f3e-47d2-adaa-8f473dec78be/volumes"
Nov 02 13:15:24.029722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.029693    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20982710-74e3-4acb-af83-0f1d76338674-maas-api-service-ca" pod "20982710-74e3-4acb-af83-0f1d76338674" (UID: "20982710-74e3-4acb-af83-0f1d76338674"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:24.029837 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.029822    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfv62\" (UniqueName: \"kubernetes.io/projected/e9110a44-4129-4c65-9de1-f2574890c394-kube-api-access-nfv62\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:24.032286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.032257    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20982710-74e3-4acb-af83-0f1d76338674-kube-api-access-hcl8h" pod "20982710-74e3-4acb-af83-0f1d76338674" (UID: "20982710-74e3-4acb-af83-0f1d76338674"). InnerVolumeSpecName "kube-api-access-hcl8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:24.136799 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.136591    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcl8h\" (UniqueName: \"kubernetes.io/projected/20982710-74e3-4acb-af83-0f1d76338674-kube-api-access-hcl8h\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:24.136799 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.136621    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20982710-74e3-4acb-af83-0f1d76338674-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:24.469915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.469887    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"
Nov 02 13:15:24.470118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.469886    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg" event={"ID":"20982710-74e3-4acb-af83-0f1d76338674","Type":"ContainerDied","Data":"5cc8035017793d5ddceff123f708f31c4c0816d4f063a546434923439fd77c3a"}
Nov 02 13:15:24.470118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.470096    5879 scope.go:122] "RemoveContainer" containerID="76903ad7de711f054d8b900cce2adb2f16ffc0264520dc32f94cdc2cb5e6d0b9"
Nov 02 13:15:24.473583 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.473554    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"
Nov 02 13:15:24.473583 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.473561    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv" event={"ID":"e9110a44-4129-4c65-9de1-f2574890c394","Type":"ContainerDied","Data":"fb7cf0e6b4716e7b0d6e33941f724a9cf0b0bacd95d1654941cab61680e641c7"}
Nov 02 13:15:24.484503 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.484473    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"
Nov 02 13:15:24.484503 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.484483    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw" event={"ID":"4863e18b-0942-4e01-94e1-293fd89aa56f","Type":"ContainerDied","Data":"141b33d841cf3b6c4b1fd1eb35f2a16d701a36849442005c0cba7b3626f25a4b"}
Nov 02 13:15:24.485592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.485570    5879 scope.go:122] "RemoveContainer" containerID="bca019f2c127c76aab7af8c2770affa7065b428337ef07af35fd434cf3db6d12"
Nov 02 13:15:24.500517 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.500498    5879 scope.go:122] "RemoveContainer" containerID="daf0947be532ecf57dedc1f21b30c90e95f5507341cb208dbc904dba039cc924"
Nov 02 13:15:24.504959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.504931    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"]
Nov 02 13:15:24.606980 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.606951    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"]
Nov 02 13:15:24.613659 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.613625    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-035dac-29893755-hrkmv"]
Nov 02 13:15:24.649553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.649504    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"]
Nov 02 13:15:24.657029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.657002    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893755-579sg"]
Nov 02 13:15:24.699607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.699572    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"
Nov 02 13:15:24.738947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.738919    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"]
Nov 02 13:15:24.743579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.743547    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893755-dz8cw"]
Nov 02 13:15:24.756728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.756699    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"]
Nov 02 13:15:24.785258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.785231    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"
Nov 02 13:15:24.790873 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.790847    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"
Nov 02 13:15:24.855600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.852768    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97847d4-325d-4ce4-874d-199279b16a87-kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/e97847d4-325d-4ce4-874d-199279b16a87-kube-api-access-5th4k\") pod \"e97847d4-325d-4ce4-874d-199279b16a87\" (UID: \"e97847d4-325d-4ce4-874d-199279b16a87\") "
Nov 02 13:15:24.855600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.853858    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-kube-api-access-bfwdf\" (UniqueName: \"kubernetes.io/projected/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-kube-api-access-bfwdf\") pod \"9025c2c7-57b9-439c-bbf8-91a5fd1433c0\" (UID: \"9025c2c7-57b9-439c-bbf8-91a5fd1433c0\") "
Nov 02 13:15:24.855600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.853946    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-maas-api-service-ca\") pod \"9025c2c7-57b9-439c-bbf8-91a5fd1433c0\" (UID: \"9025c2c7-57b9-439c-bbf8-91a5fd1433c0\") "
Nov 02 13:15:24.855600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.854369    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-maas-api-service-ca" pod "9025c2c7-57b9-439c-bbf8-91a5fd1433c0" (UID: "9025c2c7-57b9-439c-bbf8-91a5fd1433c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:24.855600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.854778    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97847d4-325d-4ce4-874d-199279b16a87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97847d4-325d-4ce4-874d-199279b16a87-maas-api-service-ca\") pod \"e97847d4-325d-4ce4-874d-199279b16a87\" (UID: \"e97847d4-325d-4ce4-874d-199279b16a87\") "
Nov 02 13:15:24.855600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.855263    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97847d4-325d-4ce4-874d-199279b16a87-maas-api-service-ca" pod "e97847d4-325d-4ce4-874d-199279b16a87" (UID: "e97847d4-325d-4ce4-874d-199279b16a87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:24.855924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.855664    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97847d4-325d-4ce4-874d-199279b16a87-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:24.855924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.855688    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:24.856471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.856397    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97847d4-325d-4ce4-874d-199279b16a87-kube-api-access-5th4k" pod "e97847d4-325d-4ce4-874d-199279b16a87" (UID: "e97847d4-325d-4ce4-874d-199279b16a87"). InnerVolumeSpecName "kube-api-access-5th4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:24.858212 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.858188    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-kube-api-access-bfwdf" pod "9025c2c7-57b9-439c-bbf8-91a5fd1433c0" (UID: "9025c2c7-57b9-439c-bbf8-91a5fd1433c0"). InnerVolumeSpecName "kube-api-access-bfwdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:24.930249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.929862    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"
Nov 02 13:15:24.957721 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.957678    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8964b32-2de3-4bb6-b25c-7d7a561d0251-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8964b32-2de3-4bb6-b25c-7d7a561d0251-maas-api-service-ca\") pod \"d8964b32-2de3-4bb6-b25c-7d7a561d0251\" (UID: \"d8964b32-2de3-4bb6-b25c-7d7a561d0251\") "
Nov 02 13:15:24.957912 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.957894    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8964b32-2de3-4bb6-b25c-7d7a561d0251-kube-api-access-ktq7m\" (UniqueName: \"kubernetes.io/projected/d8964b32-2de3-4bb6-b25c-7d7a561d0251-kube-api-access-ktq7m\") pod \"d8964b32-2de3-4bb6-b25c-7d7a561d0251\" (UID: \"d8964b32-2de3-4bb6-b25c-7d7a561d0251\") "
Nov 02 13:15:24.958578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.958554    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5th4k\" (UniqueName: \"kubernetes.io/projected/e97847d4-325d-4ce4-874d-199279b16a87-kube-api-access-5th4k\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:24.958627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.958586    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfwdf\" (UniqueName: \"kubernetes.io/projected/9025c2c7-57b9-439c-bbf8-91a5fd1433c0-kube-api-access-bfwdf\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:24.959263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.959238    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8964b32-2de3-4bb6-b25c-7d7a561d0251-maas-api-service-ca" pod "d8964b32-2de3-4bb6-b25c-7d7a561d0251" (UID: "d8964b32-2de3-4bb6-b25c-7d7a561d0251"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:24.961448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:24.961424    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8964b32-2de3-4bb6-b25c-7d7a561d0251-kube-api-access-ktq7m" pod "d8964b32-2de3-4bb6-b25c-7d7a561d0251" (UID: "d8964b32-2de3-4bb6-b25c-7d7a561d0251"). InnerVolumeSpecName "kube-api-access-ktq7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:25.061068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.059864    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031e33ac-9ee4-4153-9bdb-5289b569a146-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031e33ac-9ee4-4153-9bdb-5289b569a146-maas-api-service-ca\") pod \"031e33ac-9ee4-4153-9bdb-5289b569a146\" (UID: \"031e33ac-9ee4-4153-9bdb-5289b569a146\") "
Nov 02 13:15:25.061068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.060118    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031e33ac-9ee4-4153-9bdb-5289b569a146-kube-api-access-x2nc5\" (UniqueName: \"kubernetes.io/projected/031e33ac-9ee4-4153-9bdb-5289b569a146-kube-api-access-x2nc5\") pod \"031e33ac-9ee4-4153-9bdb-5289b569a146\" (UID: \"031e33ac-9ee4-4153-9bdb-5289b569a146\") "
Nov 02 13:15:25.061068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.060764    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8964b32-2de3-4bb6-b25c-7d7a561d0251-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:25.061068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.060783    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktq7m\" (UniqueName: \"kubernetes.io/projected/d8964b32-2de3-4bb6-b25c-7d7a561d0251-kube-api-access-ktq7m\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:25.061764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.061734    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031e33ac-9ee4-4153-9bdb-5289b569a146-maas-api-service-ca" pod "031e33ac-9ee4-4153-9bdb-5289b569a146" (UID: "031e33ac-9ee4-4153-9bdb-5289b569a146"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:15:25.065739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.065711    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031e33ac-9ee4-4153-9bdb-5289b569a146-kube-api-access-x2nc5" pod "031e33ac-9ee4-4153-9bdb-5289b569a146" (UID: "031e33ac-9ee4-4153-9bdb-5289b569a146"). InnerVolumeSpecName "kube-api-access-x2nc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:15:25.161878 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.161847    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031e33ac-9ee4-4153-9bdb-5289b569a146-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:25.161878 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.161872    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2nc5\" (UniqueName: \"kubernetes.io/projected/031e33ac-9ee4-4153-9bdb-5289b569a146-kube-api-access-x2nc5\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:15:25.452534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.452492    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:25.499191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.499156    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"
Nov 02 13:15:25.499378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.499152    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82" event={"ID":"e97847d4-325d-4ce4-874d-199279b16a87","Type":"ContainerDied","Data":"7eb781d193fd948a5a983c132aa545d836f84ec06eea29c213039a277acd8def"}
Nov 02 13:15:25.499378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.499342    5879 scope.go:122] "RemoveContainer" containerID="75e8e7bf93ecc05ba1dc9ecf27fbd1891e5d14ae02787d29dc9b447c349b4770"
Nov 02 13:15:25.508717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.508681    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c" event={"ID":"d8964b32-2de3-4bb6-b25c-7d7a561d0251","Type":"ContainerDied","Data":"dab19a4095d363b0727b081d93d4d44a2a49c4248a5c72b1306636efeb7fb6a6"}
Nov 02 13:15:25.508905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.508798    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"
Nov 02 13:15:25.514525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.514490    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt" event={"ID":"9025c2c7-57b9-439c-bbf8-91a5fd1433c0","Type":"ContainerDied","Data":"6ddb8498ebb26b073094d5cafb0348df67be139f0b1be1e26ed5ff901573556f"}
Nov 02 13:15:25.514704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.514611    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"
Nov 02 13:15:25.524419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.524393    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"
Nov 02 13:15:25.524575 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.524467    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2" event={"ID":"031e33ac-9ee4-4153-9bdb-5289b569a146","Type":"ContainerDied","Data":"ec80af19e32f2b893643f2a1065f9276a6174bd093ffb46a06e39e9debff7b37"}
Nov 02 13:15:25.526815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.526534    5879 scope.go:122] "RemoveContainer" containerID="4cbdc141c97883c4ed10f25a43e24b49935173a78df80f87e3e51af2c737d2e6"
Nov 02 13:15:25.539756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.539732    5879 scope.go:122] "RemoveContainer" containerID="fbe8b466c54dc29ffb9fa3ddc292d73252b9db6008b652f84e4243f50d068081"
Nov 02 13:15:25.550889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.550871    5879 scope.go:122] "RemoveContainer" containerID="9b30cadde4342f4abf843fd13ae7e568c8f599ae5ac9b0f031abc69b2108ac6b"
Nov 02 13:15:25.596927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.596893    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"]
Nov 02 13:15:25.600975 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.600952    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893755-v7cxt"]
Nov 02 13:15:25.639299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.639264    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"]
Nov 02 13:15:25.643259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.643233    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-8032f3-29893755-fw7f2"]
Nov 02 13:15:25.682308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.682264    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"]
Nov 02 13:15:25.686038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.686017    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-1aa18c-29893755-67j82"]
Nov 02 13:15:25.730821 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.730745    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"]
Nov 02 13:15:25.735889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:25.735861    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893755-cbq8c"]
Nov 02 13:15:26.020020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:26.019917    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146" path="/var/lib/kubelet/pods/031e33ac-9ee4-4153-9bdb-5289b569a146/volumes"
Nov 02 13:15:26.020469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:26.020455    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20982710-74e3-4acb-af83-0f1d76338674" path="/var/lib/kubelet/pods/20982710-74e3-4acb-af83-0f1d76338674/volumes"
Nov 02 13:15:26.020843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:26.020831    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f" path="/var/lib/kubelet/pods/4863e18b-0942-4e01-94e1-293fd89aa56f/volumes"
Nov 02 13:15:26.021577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:26.021562    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" path="/var/lib/kubelet/pods/9025c2c7-57b9-439c-bbf8-91a5fd1433c0/volumes"
Nov 02 13:15:26.021951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:26.021939    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" path="/var/lib/kubelet/pods/d8964b32-2de3-4bb6-b25c-7d7a561d0251/volumes"
Nov 02 13:15:26.022367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:26.022354    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9110a44-4129-4c65-9de1-f2574890c394" path="/var/lib/kubelet/pods/e9110a44-4129-4c65-9de1-f2574890c394/volumes"
Nov 02 13:15:26.023030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:26.023017    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97847d4-325d-4ce4-874d-199279b16a87" path="/var/lib/kubelet/pods/e97847d4-325d-4ce4-874d-199279b16a87/volumes"
Nov 02 13:15:27.046973 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:27.046935    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:28.737115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:28.737080    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:30.444488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:30.444438    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:31.976716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:31.976673    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:33.753182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:33.753139    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:35.489150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:35.489116    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:35.629505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:35.629465    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:35.874793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:35.874744    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:36.840871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:36.840837    5879 scope.go:122] "RemoveContainer" containerID="0f273e2670dda97528850879e8273f86933cf1fed880129204cac533a18bafba"
Nov 02 13:15:36.852763 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:36.852734    5879 scope.go:122] "RemoveContainer" containerID="948c1160154dcba48b4c3e28b947651d75f9d7791839612ff4b988704bd4d1d5"
Nov 02 13:15:36.863958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:36.863936    5879 scope.go:122] "RemoveContainer" containerID="6d8a60214ee5f0000ed27f14658ebbcb3955716993236f0580f5f90d82f85ac6"
Nov 02 13:15:36.874852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:36.874828    5879 scope.go:122] "RemoveContainer" containerID="9f096cb35bb6353626d186628134ed3969a796f8bc165fc83506a2d3f2626ec8"
Nov 02 13:15:36.885981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:36.885959    5879 scope.go:122] "RemoveContainer" containerID="221d91b913edfa572046c312f57c2745aaf9b11c379a90e0a92d36f9f8f74cb1"
Nov 02 13:15:36.896532 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:36.896507    5879 scope.go:122] "RemoveContainer" containerID="ebe5152b5dc9fa561079258f749e376aec7a02c81b8bee8e9df63895b4c7a139"
Nov 02 13:15:36.913145 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:36.913123    5879 scope.go:122] "RemoveContainer" containerID="c85f42313c65909f02188262e16d9eb984eaf2add83d64c6e40fbf3015de0303"
Nov 02 13:15:37.125823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:37.125771    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:38.776390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:38.776353    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:39.314850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:39.314803    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:40.357343 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:40.357293    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:41.995770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:41.995739    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:43.634040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:43.633948    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:44.823948 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:15:44.823911    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:15:45.452843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:45.452808    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:47.144286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:47.144250    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:48.819381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:48.819347    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:50.394987 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:50.394951    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:51.952038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:51.951986    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:53.583800 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:53.583744    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:55.136381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:55.136336    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:55.262032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:55.261971    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:56.842055 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:56.842013    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:15:58.431769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:15:58.431728    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:00.148867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:00.148831    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:01.810205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:01.810161    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:03.383067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:03.382968    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:05.066395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:05.066356    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:06.032543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:06.032509    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"]
Nov 02 13:16:06.046469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:06.046428    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ff2a4e6-mp6v9"]
Nov 02 13:16:06.588680 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:06.588648    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:08.020365 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:08.020326    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a977c9b7-c0c7-4aeb-a5f2-53c190688e50" path="/var/lib/kubelet/pods/a977c9b7-c0c7-4aeb-a5f2-53c190688e50/volumes"
Nov 02 13:16:08.142154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:08.142114    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:09.746719 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:09.746682    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:11.347654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:11.347616    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:13.048892 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:13.048851    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:14.668026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:14.667958    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:14.822181 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:16:14.822147    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:16:16.335548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:16.335514    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:17.887765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:17.887730    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:19.409111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:19.409072    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:21.247305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:21.247269    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:22.875024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:22.871180    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:24.884826 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:24.884797    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:25.015910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:25.015883    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:26.056521 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:26.056490    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:27.705267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:27.705234    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:28.532374 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:28.532339    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:29.488088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.488052    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:29.816494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.816416    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"]
Nov 02 13:16:29.817315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817289    5879 cpu_manager.go:414] "containerMap: removing container" podUID="23754db5-04c9-48a5-841a-433830943be2" containerName="cleanup"
Nov 02 13:16:29.817315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817314    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23754db5-04c9-48a5-841a-433830943be2" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817325    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817331    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817339    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817343    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817352    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4c723d82-c700-4def-b8c0-081c5e358b18" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817355    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c723d82-c700-4def-b8c0-081c5e358b18" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817362    5879 cpu_manager.go:414] "containerMap: removing container" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817366    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817372    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817377    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817385    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817389    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817394    5879 cpu_manager.go:414] "containerMap: removing container" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817398    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817405    5879 cpu_manager.go:414] "containerMap: removing container" podUID="021f486a-e073-4944-8b61-ecece24ff7fe" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817410    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021f486a-e073-4944-8b61-ecece24ff7fe" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817417    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817421    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817430    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817434    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817443    5879 cpu_manager.go:414] "containerMap: removing container" podUID="265f8052-98a4-44a8-95ef-239d37e81c43" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817447    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265f8052-98a4-44a8-95ef-239d37e81c43" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817456    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817460    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817468    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f" containerName="cleanup"
Nov 02 13:16:29.817464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817472    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817478    5879 cpu_manager.go:414] "containerMap: removing container" podUID="20982710-74e3-4acb-af83-0f1d76338674" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817482    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20982710-74e3-4acb-af83-0f1d76338674" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817490    5879 cpu_manager.go:414] "containerMap: removing container" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817494    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817500    5879 cpu_manager.go:414] "containerMap: removing container" podUID="23754db5-04c9-48a5-841a-433830943be2" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817504    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23754db5-04c9-48a5-841a-433830943be2" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817510    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817514    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817523    5879 cpu_manager.go:414] "containerMap: removing container" podUID="20982710-74e3-4acb-af83-0f1d76338674" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817527    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20982710-74e3-4acb-af83-0f1d76338674" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817532    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817536    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817545    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817549    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817558    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e97847d4-325d-4ce4-874d-199279b16a87" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817562    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97847d4-325d-4ce4-874d-199279b16a87" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817569    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817572    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817577    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817582    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817588    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4c723d82-c700-4def-b8c0-081c5e358b18" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817593    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c723d82-c700-4def-b8c0-081c5e358b18" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817600    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e97847d4-325d-4ce4-874d-199279b16a87" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817604    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97847d4-325d-4ce4-874d-199279b16a87" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817612    5879 cpu_manager.go:414] "containerMap: removing container" podUID="265f8052-98a4-44a8-95ef-239d37e81c43" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817616    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265f8052-98a4-44a8-95ef-239d37e81c43" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817623    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e97847d4-325d-4ce4-874d-199279b16a87" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817626    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97847d4-325d-4ce4-874d-199279b16a87" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817634    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4c723d82-c700-4def-b8c0-081c5e358b18" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817637    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c723d82-c700-4def-b8c0-081c5e358b18" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817646    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817651    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817661    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817665    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817671    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817675    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817682    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e9110a44-4129-4c65-9de1-f2574890c394" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817686    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9110a44-4129-4c65-9de1-f2574890c394" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817695    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e9110a44-4129-4c65-9de1-f2574890c394" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817700    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9110a44-4129-4c65-9de1-f2574890c394" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817709    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817713    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817721    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817725    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817735    5879 cpu_manager.go:414] "containerMap: removing container" podUID="021f486a-e073-4944-8b61-ecece24ff7fe" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817739    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021f486a-e073-4944-8b61-ecece24ff7fe" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817749    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817754    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817764    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817768    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817777    5879 cpu_manager.go:414] "containerMap: removing container" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817782    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817790    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817794    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817804    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817808    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817926    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c723d82-c700-4def-b8c0-081c5e358b18" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817934    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817940    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="20982710-74e3-4acb-af83-0f1d76338674" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817946    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817951    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817958    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817963    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9110a44-4129-4c65-9de1-f2574890c394" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817971    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817975    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9110a44-4129-4c65-9de1-f2574890c394" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817981    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="23754db5-04c9-48a5-841a-433830943be2" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.817985    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818007    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818012    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818018    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818025    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="021f486a-e073-4944-8b61-ecece24ff7fe" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818030    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="021f486a-e073-4944-8b61-ecece24ff7fe" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818037    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818042    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818047    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9110a44-4129-4c65-9de1-f2574890c394" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818053    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818057    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818062    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="265f8052-98a4-44a8-95ef-239d37e81c43" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818067    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="23754db5-04c9-48a5-841a-433830943be2" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818073    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818079    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818084    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818089    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818094    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818099    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818104    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818110    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="265f8052-98a4-44a8-95ef-239d37e81c43" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818116    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818123    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4863e18b-0942-4e01-94e1-293fd89aa56f" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818127    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818134    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818138    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="20982710-74e3-4acb-af83-0f1d76338674" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818145    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97847d4-325d-4ce4-874d-199279b16a87" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818150    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818155    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c723d82-c700-4def-b8c0-081c5e358b18" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818160    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97847d4-325d-4ce4-874d-199279b16a87" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818167    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" containerName="cleanup"
Nov 02 13:16:29.818319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.818172    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97847d4-325d-4ce4-874d-199279b16a87" containerName="cleanup"
Nov 02 13:16:29.822147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.822124    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:29.845378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.845351    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"]
Nov 02 13:16:29.854709 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.854668    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6563ed\"/\"default-dockercfg-csmbj\""
Nov 02 13:16:29.854852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.854806    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6563ed\"/\"test-model-6563ed-kserve-self-signed-certs\""
Nov 02 13:16:29.854852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.854841    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6563ed\"/\"openshift-service-ca.crt\""
Nov 02 13:16:29.854960 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.854866    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6563ed\"/\"kube-root-ca.crt\""
Nov 02 13:16:29.897167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.897129    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:29.925114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.925087    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-dshm\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:29.925248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.925195    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-tmp-dir\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:29.925248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.925228    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-model-cache\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:29.925378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.925340    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b74e33e-5277-41e1-a362-210e3d888b2e-tls-certs\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:29.925434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.925410    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb86j\" (UniqueName: \"kubernetes.io/projected/9b74e33e-5277-41e1-a362-210e3d888b2e-kube-api-access-sb86j\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:29.925599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:29.925577    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-home\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.026579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.026548    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb86j\" (UniqueName: \"kubernetes.io/projected/9b74e33e-5277-41e1-a362-210e3d888b2e-kube-api-access-sb86j\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.026789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.026688    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-home\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.026789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.026774    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-dshm\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.026931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.026819    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-tmp-dir\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.026931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.026854    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-model-cache\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.026931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.026882    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b74e33e-5277-41e1-a362-210e3d888b2e-tls-certs\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.027154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.027139    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-home\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.027245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.027227    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-tmp-dir\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.027384 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.027361    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-model-cache\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.029618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.029585    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b74e33e-5277-41e1-a362-210e3d888b2e-tls-certs\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.029766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.029648    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-dshm\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.059246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.059212    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb86j\" (UniqueName: \"kubernetes.io/projected/9b74e33e-5277-41e1-a362-210e3d888b2e-kube-api-access-sb86j\") pod \"test-model-6563ed-kserve-546b9748d5-xgqz2\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") " pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.147430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.143496    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:30.487605 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:16:30.487543    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b74e33e_5277_41e1_a362_210e3d888b2e.slice/crio-f7c3d703afb90de6e5e92eb7be07230b7631db3354442d9ef5ffa2e893e461a1 WatchSource:0}: Error finding container f7c3d703afb90de6e5e92eb7be07230b7631db3354442d9ef5ffa2e893e461a1: Status 404 returned error can't find the container with id f7c3d703afb90de6e5e92eb7be07230b7631db3354442d9ef5ffa2e893e461a1
Nov 02 13:16:30.489639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:30.489605    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"]
Nov 02 13:16:31.126389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:31.126358    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:31.157290 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:31.157258    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2" event={"ID":"9b74e33e-5277-41e1-a362-210e3d888b2e","Type":"ContainerStarted","Data":"47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975"}
Nov 02 13:16:31.157451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:31.157300    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2" event={"ID":"9b74e33e-5277-41e1-a362-210e3d888b2e","Type":"ContainerStarted","Data":"f7c3d703afb90de6e5e92eb7be07230b7631db3354442d9ef5ffa2e893e461a1"}
Nov 02 13:16:31.225017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:31.208490    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2" podStartSLOduration=2.20847107 podStartE2EDuration="2.20847107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:16:31.204072028 +0000 UTC m=+2937.310774876" watchObservedRunningTime="2026-11-02 13:16:31.20847107 +0000 UTC m=+2937.315173921"
Nov 02 13:16:32.144825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:32.144793    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:32.779929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:32.779898    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:34.740572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:34.740538    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:35.166580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.166540    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"]
Nov 02 13:16:35.167828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167806    5879 cpu_manager.go:414] "containerMap: removing container" podUID="021f486a-e073-4944-8b61-ecece24ff7fe" containerName="cleanup"
Nov 02 13:16:35.167882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167833    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021f486a-e073-4944-8b61-ecece24ff7fe" containerName="cleanup"
Nov 02 13:16:35.167882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167868    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" containerName="cleanup"
Nov 02 13:16:35.167882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167877    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e1b6178-a8b0-4b37-ae61-e2ce108e7a05" containerName="cleanup"
Nov 02 13:16:35.168007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167887    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" containerName="cleanup"
Nov 02 13:16:35.168007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167895    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" containerName="cleanup"
Nov 02 13:16:35.168007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167912    5879 cpu_manager.go:414] "containerMap: removing container" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" containerName="cleanup"
Nov 02 13:16:35.168007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167919    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397a2f80-d67d-43d4-a4f8-35f6e93e111e" containerName="cleanup"
Nov 02 13:16:35.168007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167930    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" containerName="cleanup"
Nov 02 13:16:35.168007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167937    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8964b32-2de3-4bb6-b25c-7d7a561d0251" containerName="cleanup"
Nov 02 13:16:35.168007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167950    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" containerName="cleanup"
Nov 02 13:16:35.168007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167957    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" containerName="cleanup"
Nov 02 13:16:35.168007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167968    5879 cpu_manager.go:414] "containerMap: removing container" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146" containerName="cleanup"
Nov 02 13:16:35.168007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.167975    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146" containerName="cleanup"
Nov 02 13:16:35.168007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168009    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168016    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168040    5879 cpu_manager.go:414] "containerMap: removing container" podUID="23754db5-04c9-48a5-841a-433830943be2" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168046    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23754db5-04c9-48a5-841a-433830943be2" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168064    5879 cpu_manager.go:414] "containerMap: removing container" podUID="20982710-74e3-4acb-af83-0f1d76338674" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168072    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20982710-74e3-4acb-af83-0f1d76338674" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168099    5879 cpu_manager.go:414] "containerMap: removing container" podUID="265f8052-98a4-44a8-95ef-239d37e81c43" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168105    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265f8052-98a4-44a8-95ef-239d37e81c43" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168136    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e9110a44-4129-4c65-9de1-f2574890c394" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168145    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9110a44-4129-4c65-9de1-f2574890c394" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168314    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c723d82-c700-4def-b8c0-081c5e358b18" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168326    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="20982710-74e3-4acb-af83-0f1d76338674" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168339    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="031e33ac-9ee4-4153-9bdb-5289b569a146" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168348    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9262bfd3-6bbc-4161-afa6-93fd1a6639a1" containerName="cleanup"
Nov 02 13:16:35.168359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168359    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c34bf9c9-3f3e-47d2-adaa-8f473dec78be" containerName="cleanup"
Nov 02 13:16:35.168833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168369    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="021f486a-e073-4944-8b61-ecece24ff7fe" containerName="cleanup"
Nov 02 13:16:35.168833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168383    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e518dcfa-edd5-4c9c-b128-3504862ab477" containerName="cleanup"
Nov 02 13:16:35.168833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168391    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="265f8052-98a4-44a8-95ef-239d37e81c43" containerName="cleanup"
Nov 02 13:16:35.168833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168402    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="23754db5-04c9-48a5-841a-433830943be2" containerName="cleanup"
Nov 02 13:16:35.168833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168412    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ffec64c-ff1f-42df-83a2-3d45751f8bb7" containerName="cleanup"
Nov 02 13:16:35.168833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168437    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9025c2c7-57b9-439c-bbf8-91a5fd1433c0" containerName="cleanup"
Nov 02 13:16:35.168833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.168445    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba4d701-2d78-40cf-bacc-8206cc16e923" containerName="cleanup"
Nov 02 13:16:35.173162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.173136    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:35.192442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.192414    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"]
Nov 02 13:16:35.204413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.204376    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:16:35.204558 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.204439    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:16:35.236122 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.236090    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q87s6\" (UniqueName: \"kubernetes.io/projected/ac0e3844-a606-4544-a56d-e243353f397e-kube-api-access-q87s6\") pod \"maas-api-revoke-sub-9817a0b2-d4p4c\" (UID: \"ac0e3844-a606-4544-a56d-e243353f397e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:35.236278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.236259    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9817a0b2-d4p4c\" (UID: \"ac0e3844-a606-4544-a56d-e243353f397e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:35.236564 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.236541    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9817a0b2-d4p4c\" (UID: \"ac0e3844-a606-4544-a56d-e243353f397e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:35.337847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.337811    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9817a0b2-d4p4c\" (UID: \"ac0e3844-a606-4544-a56d-e243353f397e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:35.338059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.337924    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q87s6\" (UniqueName: \"kubernetes.io/projected/ac0e3844-a606-4544-a56d-e243353f397e-kube-api-access-q87s6\") pod \"maas-api-revoke-sub-9817a0b2-d4p4c\" (UID: \"ac0e3844-a606-4544-a56d-e243353f397e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:35.338059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.338010    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9817a0b2-d4p4c\" (UID: \"ac0e3844-a606-4544-a56d-e243353f397e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:35.338488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.338465    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9817a0b2-d4p4c\" (UID: \"ac0e3844-a606-4544-a56d-e243353f397e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:35.340582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.340557    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9817a0b2-d4p4c\" (UID: \"ac0e3844-a606-4544-a56d-e243353f397e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:35.363507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.363475    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q87s6\" (UniqueName: \"kubernetes.io/projected/ac0e3844-a606-4544-a56d-e243353f397e-kube-api-access-q87s6\") pod \"maas-api-revoke-sub-9817a0b2-d4p4c\" (UID: \"ac0e3844-a606-4544-a56d-e243353f397e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:35.494454 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.494374    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:35.822916 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:16:35.822886    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0e3844_a606_4544_a56d_e243353f397e.slice/crio-81e3fe6c1c790cc0713ecaeae53049805258bb7caf50d78f4956194169a0d911 WatchSource:0}: Error finding container 81e3fe6c1c790cc0713ecaeae53049805258bb7caf50d78f4956194169a0d911: Status 404 returned error can't find the container with id 81e3fe6c1c790cc0713ecaeae53049805258bb7caf50d78f4956194169a0d911
Nov 02 13:16:35.825242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:35.825215    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"]
Nov 02 13:16:36.203061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:36.203027    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c" event={"ID":"ac0e3844-a606-4544-a56d-e243353f397e","Type":"ContainerStarted","Data":"2cae1efd8435b71b880ef9c00ae5bd7a5b74daad2eda56567fd3478b25a3e272"}
Nov 02 13:16:36.203195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:36.203095    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c" event={"ID":"ac0e3844-a606-4544-a56d-e243353f397e","Type":"ContainerStarted","Data":"81e3fe6c1c790cc0713ecaeae53049805258bb7caf50d78f4956194169a0d911"}
Nov 02 13:16:36.246891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:36.246843    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c" podStartSLOduration=1.246830814 podStartE2EDuration="1.246830814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:16:36.245986558 +0000 UTC m=+2942.352689409" watchObservedRunningTime="2026-11-02 13:16:36.246830814 +0000 UTC m=+2942.353533664"
Nov 02 13:16:36.390248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:36.390208    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:37.095443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:37.095369    5879 scope.go:122] "RemoveContainer" containerID="e95b8bca830d4b72d0c3338d1c3c380bcff7ffceefd9ce6fcd58baae980e8771"
Nov 02 13:16:37.107572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:37.107550    5879 scope.go:122] "RemoveContainer" containerID="1607a7376542c7fa1199e5dcfda89f3de6c05b1f61543565747b6b9c33f9fb1f"
Nov 02 13:16:37.118308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:37.118280    5879 scope.go:122] "RemoveContainer" containerID="5cbe265c473543de5843a024cc0d6ffae7c01172204afdd2f39bea8a124b39f8"
Nov 02 13:16:37.218169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:37.218139    5879 generic.go:356] "Generic (PLEG): container finished" podID="ac0e3844-a606-4544-a56d-e243353f397e" containerID="2cae1efd8435b71b880ef9c00ae5bd7a5b74daad2eda56567fd3478b25a3e272" exitCode=0
Nov 02 13:16:37.218341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:37.218227    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c" event={"ID":"ac0e3844-a606-4544-a56d-e243353f397e","Type":"ContainerDied","Data":"2cae1efd8435b71b880ef9c00ae5bd7a5b74daad2eda56567fd3478b25a3e272"}
Nov 02 13:16:38.012953 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:38.012913    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:38.401903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:38.401879    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:38.482664 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:38.482639    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac0e3844-a606-4544-a56d-e243353f397e-kube-api-access-q87s6\" (UniqueName: \"kubernetes.io/projected/ac0e3844-a606-4544-a56d-e243353f397e-kube-api-access-q87s6\") pod \"ac0e3844-a606-4544-a56d-e243353f397e\" (UID: \"ac0e3844-a606-4544-a56d-e243353f397e\") "
Nov 02 13:16:38.482851 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:38.482774    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-openshift-ca\") pod \"ac0e3844-a606-4544-a56d-e243353f397e\" (UID: \"ac0e3844-a606-4544-a56d-e243353f397e\") "
Nov 02 13:16:38.482954 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:38.482939    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-xks-ca\") pod \"ac0e3844-a606-4544-a56d-e243353f397e\" (UID: \"ac0e3844-a606-4544-a56d-e243353f397e\") "
Nov 02 13:16:38.483147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:38.483121    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-openshift-ca" pod "ac0e3844-a606-4544-a56d-e243353f397e" (UID: "ac0e3844-a606-4544-a56d-e243353f397e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:16:38.483870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:38.483846    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:16:38.485277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:38.485253    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac0e3844-a606-4544-a56d-e243353f397e-kube-api-access-q87s6" pod "ac0e3844-a606-4544-a56d-e243353f397e" (UID: "ac0e3844-a606-4544-a56d-e243353f397e"). InnerVolumeSpecName "kube-api-access-q87s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:16:38.485379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:38.485334    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-xks-ca" pod "ac0e3844-a606-4544-a56d-e243353f397e" (UID: "ac0e3844-a606-4544-a56d-e243353f397e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:16:38.585208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:38.585153    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac0e3844-a606-4544-a56d-e243353f397e-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:16:38.585208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:38.585183    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q87s6\" (UniqueName: \"kubernetes.io/projected/ac0e3844-a606-4544-a56d-e243353f397e-kube-api-access-q87s6\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:16:39.234945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:39.234919    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"
Nov 02 13:16:39.234945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:39.234937    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c" event={"ID":"ac0e3844-a606-4544-a56d-e243353f397e","Type":"ContainerDied","Data":"81e3fe6c1c790cc0713ecaeae53049805258bb7caf50d78f4956194169a0d911"}
Nov 02 13:16:39.235166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:39.234978    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81e3fe6c1c790cc0713ecaeae53049805258bb7caf50d78f4956194169a0d911"
Nov 02 13:16:39.450326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:39.450290    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:39.579547 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:39.579391    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:40.144796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:40.144765    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:40.158034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:40.157981    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:40.254445 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:40.254414    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:16:41.231258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:41.231227    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:42.744593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:42.744558    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:44.255818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:44.255787    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:44.821548 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:16:44.821512    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:16:46.037833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.037803    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:46.145401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.145368    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"]
Nov 02 13:16:46.146216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.146184    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ac0e3844-a606-4544-a56d-e243353f397e" containerName="revoke-keys"
Nov 02 13:16:46.146216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.146201    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac0e3844-a606-4544-a56d-e243353f397e" containerName="revoke-keys"
Nov 02 13:16:46.146363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.146354    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac0e3844-a606-4544-a56d-e243353f397e" containerName="revoke-keys"
Nov 02 13:16:46.150867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.150803    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:46.171352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.171324    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"]
Nov 02 13:16:46.185283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.185256    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:16:46.185440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.185279    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:16:46.298824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.298740    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52e0d30e-pmz82\" (UID: \"37121a77-285c-4f67-81fa-00b4ba4b2288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:46.298985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.298923    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27mst\" (UniqueName: \"kubernetes.io/projected/37121a77-285c-4f67-81fa-00b4ba4b2288-kube-api-access-27mst\") pod \"maas-api-revoke-sub-52e0d30e-pmz82\" (UID: \"37121a77-285c-4f67-81fa-00b4ba4b2288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:46.299111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.299070    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52e0d30e-pmz82\" (UID: \"37121a77-285c-4f67-81fa-00b4ba4b2288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:46.400921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.400893    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52e0d30e-pmz82\" (UID: \"37121a77-285c-4f67-81fa-00b4ba4b2288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:46.401089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.400959    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27mst\" (UniqueName: \"kubernetes.io/projected/37121a77-285c-4f67-81fa-00b4ba4b2288-kube-api-access-27mst\") pod \"maas-api-revoke-sub-52e0d30e-pmz82\" (UID: \"37121a77-285c-4f67-81fa-00b4ba4b2288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:46.401089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.401044    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52e0d30e-pmz82\" (UID: \"37121a77-285c-4f67-81fa-00b4ba4b2288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:46.401725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.401703    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52e0d30e-pmz82\" (UID: \"37121a77-285c-4f67-81fa-00b4ba4b2288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:46.403473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.403455    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52e0d30e-pmz82\" (UID: \"37121a77-285c-4f67-81fa-00b4ba4b2288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:46.426154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.426128    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27mst\" (UniqueName: \"kubernetes.io/projected/37121a77-285c-4f67-81fa-00b4ba4b2288-kube-api-access-27mst\") pod \"maas-api-revoke-sub-52e0d30e-pmz82\" (UID: \"37121a77-285c-4f67-81fa-00b4ba4b2288\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:46.467600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.467576    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:46.781329 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:16:46.781284    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37121a77_285c_4f67_81fa_00b4ba4b2288.slice/crio-4475c3309b7b0d487c2ea72a58678cfca70a6054ce2a27235bd47e69e431f468 WatchSource:0}: Error finding container 4475c3309b7b0d487c2ea72a58678cfca70a6054ce2a27235bd47e69e431f468: Status 404 returned error can't find the container with id 4475c3309b7b0d487c2ea72a58678cfca70a6054ce2a27235bd47e69e431f468
Nov 02 13:16:46.782799 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:46.782777    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"]
Nov 02 13:16:47.323667 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:47.323632    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82" event={"ID":"37121a77-285c-4f67-81fa-00b4ba4b2288","Type":"ContainerStarted","Data":"de9532f383b97894c658f4099b7bf506b76b40f3da0d684b1b4fa57df24c5882"}
Nov 02 13:16:47.324142 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:47.323691    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82" event={"ID":"37121a77-285c-4f67-81fa-00b4ba4b2288","Type":"ContainerStarted","Data":"4475c3309b7b0d487c2ea72a58678cfca70a6054ce2a27235bd47e69e431f468"}
Nov 02 13:16:47.371412 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:47.371367    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82" podStartSLOduration=1.3713525739999999 podStartE2EDuration="1.371352574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:16:47.367020099 +0000 UTC m=+2953.473722950" watchObservedRunningTime="2026-11-02 13:16:47.371352574 +0000 UTC m=+2953.478055463"
Nov 02 13:16:47.606280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:47.606215    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:47.852433 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:47.852370    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:48.334053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:48.334017    5879 generic.go:356] "Generic (PLEG): container finished" podID="37121a77-285c-4f67-81fa-00b4ba4b2288" containerID="de9532f383b97894c658f4099b7bf506b76b40f3da0d684b1b4fa57df24c5882" exitCode=0
Nov 02 13:16:48.334505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:48.334214    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82" event={"ID":"37121a77-285c-4f67-81fa-00b4ba4b2288","Type":"ContainerDied","Data":"de9532f383b97894c658f4099b7bf506b76b40f3da0d684b1b4fa57df24c5882"}
Nov 02 13:16:49.418563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:49.418530    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:49.546890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:49.546861    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:49.659429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:49.659400    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37121a77-285c-4f67-81fa-00b4ba4b2288-kube-api-access-27mst\" (UniqueName: \"kubernetes.io/projected/37121a77-285c-4f67-81fa-00b4ba4b2288-kube-api-access-27mst\") pod \"37121a77-285c-4f67-81fa-00b4ba4b2288\" (UID: \"37121a77-285c-4f67-81fa-00b4ba4b2288\") "
Nov 02 13:16:49.659605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:49.659471    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-openshift-ca\") pod \"37121a77-285c-4f67-81fa-00b4ba4b2288\" (UID: \"37121a77-285c-4f67-81fa-00b4ba4b2288\") "
Nov 02 13:16:49.659737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:49.659723    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-xks-ca\") pod \"37121a77-285c-4f67-81fa-00b4ba4b2288\" (UID: \"37121a77-285c-4f67-81fa-00b4ba4b2288\") "
Nov 02 13:16:49.659812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:49.659788    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-openshift-ca" pod "37121a77-285c-4f67-81fa-00b4ba4b2288" (UID: "37121a77-285c-4f67-81fa-00b4ba4b2288"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:16:49.662135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:49.660803    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:16:49.665184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:49.665081    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37121a77-285c-4f67-81fa-00b4ba4b2288-kube-api-access-27mst" pod "37121a77-285c-4f67-81fa-00b4ba4b2288" (UID: "37121a77-285c-4f67-81fa-00b4ba4b2288"). InnerVolumeSpecName "kube-api-access-27mst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:16:49.676629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:49.676596    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-xks-ca" pod "37121a77-285c-4f67-81fa-00b4ba4b2288" (UID: "37121a77-285c-4f67-81fa-00b4ba4b2288"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:16:49.763672 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:49.763642    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27mst\" (UniqueName: \"kubernetes.io/projected/37121a77-285c-4f67-81fa-00b4ba4b2288-kube-api-access-27mst\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:16:49.763672 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:49.763667    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37121a77-285c-4f67-81fa-00b4ba4b2288-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:16:50.352378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:50.352302    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"
Nov 02 13:16:50.352378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:50.352339    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82" event={"ID":"37121a77-285c-4f67-81fa-00b4ba4b2288","Type":"ContainerDied","Data":"4475c3309b7b0d487c2ea72a58678cfca70a6054ce2a27235bd47e69e431f468"}
Nov 02 13:16:50.352576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:50.352387    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4475c3309b7b0d487c2ea72a58678cfca70a6054ce2a27235bd47e69e431f468"
Nov 02 13:16:51.054266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:51.054231    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:52.346221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:52.346190    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:52.686978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:52.686941    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:54.479299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:54.479253    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:56.114542 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:56.114507    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:56.419979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:56.419900    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:57.821351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:57.821313    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:58.058824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:58.058789    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"]
Nov 02 13:16:58.155906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:58.155842    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:58.275820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:58.275787    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:58.812260 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:58.812221    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:59.001746 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:59.001718    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"]
Nov 02 13:16:59.122721 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:59.122697    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:16:59.529514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:16:59.529424    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:01.148826 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:01.148795    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:02.554256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.554220    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"]
Nov 02 13:17:02.555226 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.555203    5879 cpu_manager.go:414] "containerMap: removing container" podUID="37121a77-285c-4f67-81fa-00b4ba4b2288" containerName="revoke-keys"
Nov 02 13:17:02.555226 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.555226    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37121a77-285c-4f67-81fa-00b4ba4b2288" containerName="revoke-keys"
Nov 02 13:17:02.555490 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.555475    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="37121a77-285c-4f67-81fa-00b4ba4b2288" containerName="revoke-keys"
Nov 02 13:17:02.559290 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.559258    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:02.576708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.576687    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:17:02.590687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.590664    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:17:02.598244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.598217    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"]
Nov 02 13:17:02.673374 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.673336    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99a4a0d-hskpk\" (UID: \"04769618-7fd0-4a6d-8598-93e5a118abbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:02.673536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.673496    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99a4a0d-hskpk\" (UID: \"04769618-7fd0-4a6d-8598-93e5a118abbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:02.673618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.673603    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psjcz\" (UniqueName: \"kubernetes.io/projected/04769618-7fd0-4a6d-8598-93e5a118abbc-kube-api-access-psjcz\") pod \"maas-api-revoke-sub-f99a4a0d-hskpk\" (UID: \"04769618-7fd0-4a6d-8598-93e5a118abbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:02.775230 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.775197    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99a4a0d-hskpk\" (UID: \"04769618-7fd0-4a6d-8598-93e5a118abbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:02.775403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.775319    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99a4a0d-hskpk\" (UID: \"04769618-7fd0-4a6d-8598-93e5a118abbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:02.775467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.775399    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psjcz\" (UniqueName: \"kubernetes.io/projected/04769618-7fd0-4a6d-8598-93e5a118abbc-kube-api-access-psjcz\") pod \"maas-api-revoke-sub-f99a4a0d-hskpk\" (UID: \"04769618-7fd0-4a6d-8598-93e5a118abbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:02.775786 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.775766    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f99a4a0d-hskpk\" (UID: \"04769618-7fd0-4a6d-8598-93e5a118abbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:02.777827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.777808    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f99a4a0d-hskpk\" (UID: \"04769618-7fd0-4a6d-8598-93e5a118abbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:02.801946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.801923    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psjcz\" (UniqueName: \"kubernetes.io/projected/04769618-7fd0-4a6d-8598-93e5a118abbc-kube-api-access-psjcz\") pod \"maas-api-revoke-sub-f99a4a0d-hskpk\" (UID: \"04769618-7fd0-4a6d-8598-93e5a118abbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:02.873756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.873717    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:02.945139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:02.945102    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:03.059610 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.059573    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q" podUID="0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8" containerName="payload-processing" containerID="cri-o://7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f" gracePeriod=25
Nov 02 13:17:03.213857 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:17:03.213824    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04769618_7fd0_4a6d_8598_93e5a118abbc.slice/crio-f6f8a0ff4d766de6fad0818d01051ef4c280cc7588268d3b823ee84dac0eebc1 WatchSource:0}: Error finding container f6f8a0ff4d766de6fad0818d01051ef4c280cc7588268d3b823ee84dac0eebc1: Status 404 returned error can't find the container with id f6f8a0ff4d766de6fad0818d01051ef4c280cc7588268d3b823ee84dac0eebc1
Nov 02 13:17:03.217604 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.217575    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"]
Nov 02 13:17:03.394625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.394599    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:17:03.491969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.490689    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-plugins-config-volume\") pod \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") "
Nov 02 13:17:03.491969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.490788    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-kube-api-access-72gnq\" (UniqueName: \"kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-kube-api-access-72gnq\") pod \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") "
Nov 02 13:17:03.491969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.490831    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-provider-credentials\") pod \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") "
Nov 02 13:17:03.491969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.491089    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-routing-overlay-volume\") pod \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\" (UID: \"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8\") "
Nov 02 13:17:03.492389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.492341    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-routing-overlay-volume" pod "0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8" (UID: "0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:03.493381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.492705    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-plugins-config-volume" pod "0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8" (UID: "0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:03.498014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.496250    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-kube-api-access-72gnq" pod "0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8" (UID: "0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8"). InnerVolumeSpecName "kube-api-access-72gnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:03.500810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.500778    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-provider-credentials" pod "0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8" (UID: "0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:03.503668 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.503639    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk" event={"ID":"04769618-7fd0-4a6d-8598-93e5a118abbc","Type":"ContainerStarted","Data":"f6f8a0ff4d766de6fad0818d01051ef4c280cc7588268d3b823ee84dac0eebc1"}
Nov 02 13:17:03.505509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.505485    5879 generic.go:356] "Generic (PLEG): container finished" podID="0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8" containerID="7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f" exitCode=0
Nov 02 13:17:03.505596 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.505534    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"
Nov 02 13:17:03.505661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.505599    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q" event={"ID":"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8","Type":"ContainerDied","Data":"7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f"}
Nov 02 13:17:03.505661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.505637    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q" event={"ID":"0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8","Type":"ContainerDied","Data":"31a365023d176f5a0e2dc9f454b1c8ee08de01eab29627614aefb56b4fe6e6d4"}
Nov 02 13:17:03.505743 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.505666    5879 scope.go:122] "RemoveContainer" containerID="7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f"
Nov 02 13:17:03.520252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.520232    5879 scope.go:122] "RemoveContainer" containerID="7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f"
Nov 02 13:17:03.520533 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:03.520510    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f\": container with ID starting with 7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f not found: ID does not exist" containerID="7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f"
Nov 02 13:17:03.520619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.520544    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f"} err="failed to get container status \"7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f\": rpc error: code = NotFound desc = could not find container \"7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f\": container with ID starting with 7b0a0a7bc4420c79800e4b96d0d6b52a15a379684552b6e34cfcfe5b619e8b1f not found: ID does not exist"
Nov 02 13:17:03.579785 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.579748    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"]
Nov 02 13:17:03.580272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.579809    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-14fcb6/payload-processing-external-model-e2e-extmod-a-14fcb6-dbbcwxz2q"]
Nov 02 13:17:03.594933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.594898    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:03.594933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.594928    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:03.594933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.594943    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72gnq\" (UniqueName: \"kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-kube-api-access-72gnq\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:03.595259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:03.594952    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:04.002174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.002129    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq" podUID="4c2e5684-7388-43fd-86fd-044b4e9dd54a" containerName="payload-processing" containerID="cri-o://97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18" gracePeriod=25
Nov 02 13:17:04.022475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.022447    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8" path="/var/lib/kubelet/pods/0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8/volumes"
Nov 02 13:17:04.294461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.294430    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:17:04.405477 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.405452    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-kube-api-access-gnqtp\" (UniqueName: \"kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-kube-api-access-gnqtp\") pod \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") "
Nov 02 13:17:04.405625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.405571    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-provider-credentials\") pod \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") "
Nov 02 13:17:04.405625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.405601    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-plugins-config-volume\") pod \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") "
Nov 02 13:17:04.405773 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.405757    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-routing-overlay-volume\") pod \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\" (UID: \"4c2e5684-7388-43fd-86fd-044b4e9dd54a\") "
Nov 02 13:17:04.406103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.406076    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-plugins-config-volume" pod "4c2e5684-7388-43fd-86fd-044b4e9dd54a" (UID: "4c2e5684-7388-43fd-86fd-044b4e9dd54a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:04.406235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.406188    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-routing-overlay-volume" pod "4c2e5684-7388-43fd-86fd-044b4e9dd54a" (UID: "4c2e5684-7388-43fd-86fd-044b4e9dd54a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:04.406454 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.406435    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:04.406539 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.406454    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4c2e5684-7388-43fd-86fd-044b4e9dd54a-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:04.408573 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.408549    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-kube-api-access-gnqtp" pod "4c2e5684-7388-43fd-86fd-044b4e9dd54a" (UID: "4c2e5684-7388-43fd-86fd-044b4e9dd54a"). InnerVolumeSpecName "kube-api-access-gnqtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:04.408680 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.408576    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-provider-credentials" pod "4c2e5684-7388-43fd-86fd-044b4e9dd54a" (UID: "4c2e5684-7388-43fd-86fd-044b4e9dd54a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:04.507463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.507430    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnqtp\" (UniqueName: \"kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-kube-api-access-gnqtp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:04.507463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.507456    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4c2e5684-7388-43fd-86fd-044b4e9dd54a-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:04.516687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.516656    5879 generic.go:356] "Generic (PLEG): container finished" podID="04769618-7fd0-4a6d-8598-93e5a118abbc" containerID="ef7e9e89f709027983ce0eba291fa829d60eefc191c8231e69c07956e22df6ab" exitCode=0
Nov 02 13:17:04.516834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.516723    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk" event={"ID":"04769618-7fd0-4a6d-8598-93e5a118abbc","Type":"ContainerDied","Data":"ef7e9e89f709027983ce0eba291fa829d60eefc191c8231e69c07956e22df6ab"}
Nov 02 13:17:04.519610 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.519588    5879 generic.go:356] "Generic (PLEG): container finished" podID="4c2e5684-7388-43fd-86fd-044b4e9dd54a" containerID="97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18" exitCode=0
Nov 02 13:17:04.519753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.519612    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"
Nov 02 13:17:04.519753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.519662    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq" event={"ID":"4c2e5684-7388-43fd-86fd-044b4e9dd54a","Type":"ContainerDied","Data":"97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18"}
Nov 02 13:17:04.519753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.519712    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq" event={"ID":"4c2e5684-7388-43fd-86fd-044b4e9dd54a","Type":"ContainerDied","Data":"cb2ad40c8e74719be59902deeef7c36591932ff76bbbb766cd7279afe1aa1ebd"}
Nov 02 13:17:04.519753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.519740    5879 scope.go:122] "RemoveContainer" containerID="97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18"
Nov 02 13:17:04.531362 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.531344    5879 scope.go:122] "RemoveContainer" containerID="97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18"
Nov 02 13:17:04.531625 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:04.531594    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18\": container with ID starting with 97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18 not found: ID does not exist" containerID="97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18"
Nov 02 13:17:04.531685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.531632    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18"} err="failed to get container status \"97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18\": rpc error: code = NotFound desc = could not find container \"97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18\": container with ID starting with 97b76d2d2f9ca4be4d89fc531740ec3de1d690e1d1582e9bfda3bb4f1348ca18 not found: ID does not exist"
Nov 02 13:17:04.581022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.577987    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:04.606547 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.606514    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"]
Nov 02 13:17:04.611730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:04.611701    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e3396f/payload-processing-external-model-e2e-extmod-b-e3396f-cbf7hb7mq"]
Nov 02 13:17:05.067196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.067116    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:05.224462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.218495    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:05.537812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.537783    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:05.712272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.712240    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:05.827354 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.827271    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-openshift-ca\") pod \"04769618-7fd0-4a6d-8598-93e5a118abbc\" (UID: \"04769618-7fd0-4a6d-8598-93e5a118abbc\") "
Nov 02 13:17:05.827354 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.827326    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-xks-ca\") pod \"04769618-7fd0-4a6d-8598-93e5a118abbc\" (UID: \"04769618-7fd0-4a6d-8598-93e5a118abbc\") "
Nov 02 13:17:05.827735 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.827457    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04769618-7fd0-4a6d-8598-93e5a118abbc-kube-api-access-psjcz\" (UniqueName: \"kubernetes.io/projected/04769618-7fd0-4a6d-8598-93e5a118abbc-kube-api-access-psjcz\") pod \"04769618-7fd0-4a6d-8598-93e5a118abbc\" (UID: \"04769618-7fd0-4a6d-8598-93e5a118abbc\") "
Nov 02 13:17:05.827735 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.827681    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-openshift-ca" pod "04769618-7fd0-4a6d-8598-93e5a118abbc" (UID: "04769618-7fd0-4a6d-8598-93e5a118abbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:05.828095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.828070    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:05.829934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.829906    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-xks-ca" pod "04769618-7fd0-4a6d-8598-93e5a118abbc" (UID: "04769618-7fd0-4a6d-8598-93e5a118abbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:17:05.830264 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.830246    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04769618-7fd0-4a6d-8598-93e5a118abbc-kube-api-access-psjcz" pod "04769618-7fd0-4a6d-8598-93e5a118abbc" (UID: "04769618-7fd0-4a6d-8598-93e5a118abbc"). InnerVolumeSpecName "kube-api-access-psjcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:05.949094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.945058    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psjcz\" (UniqueName: \"kubernetes.io/projected/04769618-7fd0-4a6d-8598-93e5a118abbc-kube-api-access-psjcz\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:05.949094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:05.945094    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04769618-7fd0-4a6d-8598-93e5a118abbc-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:06.034818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:06.034774    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c2e5684-7388-43fd-86fd-044b4e9dd54a" path="/var/lib/kubelet/pods/4c2e5684-7388-43fd-86fd-044b4e9dd54a/volumes"
Nov 02 13:17:06.145683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:06.145622    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:06.294655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:06.294625    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:06.542036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:06.541948    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"
Nov 02 13:17:06.542036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:06.541986    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk" event={"ID":"04769618-7fd0-4a6d-8598-93e5a118abbc","Type":"ContainerDied","Data":"f6f8a0ff4d766de6fad0818d01051ef4c280cc7588268d3b823ee84dac0eebc1"}
Nov 02 13:17:06.542220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:06.542053    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6f8a0ff4d766de6fad0818d01051ef4c280cc7588268d3b823ee84dac0eebc1"
Nov 02 13:17:06.605256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:06.605223    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:07.740931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:07.740901    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:09.558323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:09.558288    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:10.583654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:10.583606    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:10.707744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:10.707699    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:11.370869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:11.370839    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:12.288950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.288916    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:12.425744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.425708    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:12.439780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.439754    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"]
Nov 02 13:17:12.440607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.440590    5879 cpu_manager.go:414] "containerMap: removing container" podUID="0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8" containerName="payload-processing"
Nov 02 13:17:12.440661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.440610    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8" containerName="payload-processing"
Nov 02 13:17:12.440661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.440625    5879 cpu_manager.go:414] "containerMap: removing container" podUID="04769618-7fd0-4a6d-8598-93e5a118abbc" containerName="revoke-keys"
Nov 02 13:17:12.440661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.440631    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04769618-7fd0-4a6d-8598-93e5a118abbc" containerName="revoke-keys"
Nov 02 13:17:12.440766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.440668    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4c2e5684-7388-43fd-86fd-044b4e9dd54a" containerName="payload-processing"
Nov 02 13:17:12.440766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.440673    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c2e5684-7388-43fd-86fd-044b4e9dd54a" containerName="payload-processing"
Nov 02 13:17:12.440834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.440780    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aeb79cf-cecb-4c34-8fb3-f75dc5dc86f8" containerName="payload-processing"
Nov 02 13:17:12.440834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.440789    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="04769618-7fd0-4a6d-8598-93e5a118abbc" containerName="revoke-keys"
Nov 02 13:17:12.440834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.440802    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c2e5684-7388-43fd-86fd-044b4e9dd54a" containerName="payload-processing"
Nov 02 13:17:12.445963 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.445943    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"
Nov 02 13:17:12.466831 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.466801    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"]
Nov 02 13:17:12.475658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.475630    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e8a402\""
Nov 02 13:17:12.475658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.475636    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e8a402-dockercfg-hrd6j\""
Nov 02 13:17:12.598075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.573700    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx442\" (UniqueName: \"kubernetes.io/projected/c5749f06-f474-4e93-b640-8eaeadc2ec35-kube-api-access-tx442\") pod \"payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k\" (UID: \"c5749f06-f474-4e93-b640-8eaeadc2ec35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"
Nov 02 13:17:12.598075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.581935    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5749f06-f474-4e93-b640-8eaeadc2ec35-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k\" (UID: \"c5749f06-f474-4e93-b640-8eaeadc2ec35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"
Nov 02 13:17:12.642653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.642620    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"]
Nov 02 13:17:12.649167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.648412    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"
Nov 02 13:17:12.671127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.670252    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"]
Nov 02 13:17:12.686460 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.686422    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b54b7e-a38e-4899-8986-4b9d27c427a3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9\" (UID: \"f6b54b7e-a38e-4899-8986-4b9d27c427a3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"
Nov 02 13:17:12.686607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.686532    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5749f06-f474-4e93-b640-8eaeadc2ec35-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k\" (UID: \"c5749f06-f474-4e93-b640-8eaeadc2ec35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"
Nov 02 13:17:12.686607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.686601    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm5g7\" (UniqueName: \"kubernetes.io/projected/f6b54b7e-a38e-4899-8986-4b9d27c427a3-kube-api-access-fm5g7\") pod \"payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9\" (UID: \"f6b54b7e-a38e-4899-8986-4b9d27c427a3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"
Nov 02 13:17:12.686689 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.686622    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx442\" (UniqueName: \"kubernetes.io/projected/c5749f06-f474-4e93-b640-8eaeadc2ec35-kube-api-access-tx442\") pod \"payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k\" (UID: \"c5749f06-f474-4e93-b640-8eaeadc2ec35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"
Nov 02 13:17:12.687679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.687651    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5749f06-f474-4e93-b640-8eaeadc2ec35-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k\" (UID: \"c5749f06-f474-4e93-b640-8eaeadc2ec35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"
Nov 02 13:17:12.737088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.729067    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx442\" (UniqueName: \"kubernetes.io/projected/c5749f06-f474-4e93-b640-8eaeadc2ec35-kube-api-access-tx442\") pod \"payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k\" (UID: \"c5749f06-f474-4e93-b640-8eaeadc2ec35\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"
Nov 02 13:17:12.760647 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.760613    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"
Nov 02 13:17:12.787885 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.787854    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b54b7e-a38e-4899-8986-4b9d27c427a3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9\" (UID: \"f6b54b7e-a38e-4899-8986-4b9d27c427a3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"
Nov 02 13:17:12.788080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.787970    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm5g7\" (UniqueName: \"kubernetes.io/projected/f6b54b7e-a38e-4899-8986-4b9d27c427a3-kube-api-access-fm5g7\") pod \"payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9\" (UID: \"f6b54b7e-a38e-4899-8986-4b9d27c427a3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"
Nov 02 13:17:12.788572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.788548    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b54b7e-a38e-4899-8986-4b9d27c427a3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9\" (UID: \"f6b54b7e-a38e-4899-8986-4b9d27c427a3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"
Nov 02 13:17:12.813734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.813699    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm5g7\" (UniqueName: \"kubernetes.io/projected/f6b54b7e-a38e-4899-8986-4b9d27c427a3-kube-api-access-fm5g7\") pod \"payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9\" (UID: \"f6b54b7e-a38e-4899-8986-4b9d27c427a3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"
Nov 02 13:17:12.989855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:12.989817    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"
Nov 02 13:17:13.080354 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.080324    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"]
Nov 02 13:17:13.241667 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.241603    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:13.307153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.306253    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"]
Nov 02 13:17:13.308855 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:17:13.308826    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b54b7e_a38e_4899_8986_4b9d27c427a3.slice/crio-0abb5576596c1eda6541259809fc5d88e2c3e5179d8b15c4a062d22ea5f3df89 WatchSource:0}: Error finding container 0abb5576596c1eda6541259809fc5d88e2c3e5179d8b15c4a062d22ea5f3df89: Status 404 returned error can't find the container with id 0abb5576596c1eda6541259809fc5d88e2c3e5179d8b15c4a062d22ea5f3df89
Nov 02 13:17:13.351850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.351823    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"]
Nov 02 13:17:13.360556 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.360533    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:13.381861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.377165    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:17:13.381861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.377243    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:17:13.381861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.377740    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"]
Nov 02 13:17:13.399348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.399322    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b23818-fnh2j\" (UID: \"3a891d4b-f123-412f-8810-ae355a07ffbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:13.399539 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.399525    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b23818-fnh2j\" (UID: \"3a891d4b-f123-412f-8810-ae355a07ffbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:13.399618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.399606    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k4mp\" (UniqueName: \"kubernetes.io/projected/3a891d4b-f123-412f-8810-ae355a07ffbd-kube-api-access-7k4mp\") pod \"maas-api-revoke-sub-a2b23818-fnh2j\" (UID: \"3a891d4b-f123-412f-8810-ae355a07ffbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:13.501058 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.500924    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k4mp\" (UniqueName: \"kubernetes.io/projected/3a891d4b-f123-412f-8810-ae355a07ffbd-kube-api-access-7k4mp\") pod \"maas-api-revoke-sub-a2b23818-fnh2j\" (UID: \"3a891d4b-f123-412f-8810-ae355a07ffbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:13.501209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.501096    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b23818-fnh2j\" (UID: \"3a891d4b-f123-412f-8810-ae355a07ffbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:13.501398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.501380    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b23818-fnh2j\" (UID: \"3a891d4b-f123-412f-8810-ae355a07ffbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:13.501793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.501771    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b23818-fnh2j\" (UID: \"3a891d4b-f123-412f-8810-ae355a07ffbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:13.503897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.503879    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b23818-fnh2j\" (UID: \"3a891d4b-f123-412f-8810-ae355a07ffbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:13.526129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.526110    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k4mp\" (UniqueName: \"kubernetes.io/projected/3a891d4b-f123-412f-8810-ae355a07ffbd-kube-api-access-7k4mp\") pod \"maas-api-revoke-sub-a2b23818-fnh2j\" (UID: \"3a891d4b-f123-412f-8810-ae355a07ffbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:13.631261 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.631224    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k" event={"ID":"c5749f06-f474-4e93-b640-8eaeadc2ec35","Type":"ContainerStarted","Data":"3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d"}
Nov 02 13:17:13.631452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.631290    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"
Nov 02 13:17:13.631452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.631315    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k" event={"ID":"c5749f06-f474-4e93-b640-8eaeadc2ec35","Type":"ContainerStarted","Data":"60697fe279a3d529cc8d928114fe678be1ddcc923242703c2f3d0e1387bd92a0"}
Nov 02 13:17:13.633529 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.633492    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9" event={"ID":"f6b54b7e-a38e-4899-8986-4b9d27c427a3","Type":"ContainerStarted","Data":"da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8"}
Nov 02 13:17:13.633644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.633537    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9" event={"ID":"f6b54b7e-a38e-4899-8986-4b9d27c427a3","Type":"ContainerStarted","Data":"0abb5576596c1eda6541259809fc5d88e2c3e5179d8b15c4a062d22ea5f3df89"}
Nov 02 13:17:13.633644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.633556    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"
Nov 02 13:17:13.677589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.677539    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k" podStartSLOduration=1.677521761 podStartE2EDuration="1.677521761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:13.674228992 +0000 UTC m=+2979.780931844" watchObservedRunningTime="2026-11-02 13:17:13.677521761 +0000 UTC m=+2979.784224612"
Nov 02 13:17:13.706928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.706899    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:13.721286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:13.721240    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9" podStartSLOduration=1.721226419 podStartE2EDuration="1.721226419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:13.71820538 +0000 UTC m=+2979.824908232" watchObservedRunningTime="2026-11-02 13:17:13.721226419 +0000 UTC m=+2979.827929271"
Nov 02 13:17:14.027363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:14.027336    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"]
Nov 02 13:17:14.643410 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:14.643306    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j" event={"ID":"3a891d4b-f123-412f-8810-ae355a07ffbd","Type":"ContainerStarted","Data":"a4cbcdf1f44951272f6fbeb94f1b30c317063b603dd3c75c4f220e5759e7d064"}
Nov 02 13:17:14.643410 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:14.643371    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j" event={"ID":"3a891d4b-f123-412f-8810-ae355a07ffbd","Type":"ContainerStarted","Data":"9175456c8b856bbc2b760bf991c8bd3d00e2d8fd14b45b48049b39bdf656a7fe"}
Nov 02 13:17:14.689876 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:14.689823    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j" podStartSLOduration=1.689809038 podStartE2EDuration="1.689809038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:14.687447832 +0000 UTC m=+2980.794150684" watchObservedRunningTime="2026-11-02 13:17:14.689809038 +0000 UTC m=+2980.796511921"
Nov 02 13:17:14.821861 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:14.821821    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:17:14.899524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:14.899427    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:15.655819 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:15.655761    5879 generic.go:356] "Generic (PLEG): container finished" podID="3a891d4b-f123-412f-8810-ae355a07ffbd" containerID="a4cbcdf1f44951272f6fbeb94f1b30c317063b603dd3c75c4f220e5759e7d064" exitCode=0
Nov 02 13:17:15.656373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:15.655874    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j" event={"ID":"3a891d4b-f123-412f-8810-ae355a07ffbd","Type":"ContainerDied","Data":"a4cbcdf1f44951272f6fbeb94f1b30c317063b603dd3c75c4f220e5759e7d064"}
Nov 02 13:17:16.578940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:16.578904    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:16.842234 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:16.842164    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:16.885549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:16.885506    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-openshift-ca\") pod \"3a891d4b-f123-412f-8810-ae355a07ffbd\" (UID: \"3a891d4b-f123-412f-8810-ae355a07ffbd\") "
Nov 02 13:17:16.885747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:16.885578    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-xks-ca\") pod \"3a891d4b-f123-412f-8810-ae355a07ffbd\" (UID: \"3a891d4b-f123-412f-8810-ae355a07ffbd\") "
Nov 02 13:17:16.885747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:16.885695    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a891d4b-f123-412f-8810-ae355a07ffbd-kube-api-access-7k4mp\" (UniqueName: \"kubernetes.io/projected/3a891d4b-f123-412f-8810-ae355a07ffbd-kube-api-access-7k4mp\") pod \"3a891d4b-f123-412f-8810-ae355a07ffbd\" (UID: \"3a891d4b-f123-412f-8810-ae355a07ffbd\") "
Nov 02 13:17:16.885985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:16.885962    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-openshift-ca" pod "3a891d4b-f123-412f-8810-ae355a07ffbd" (UID: "3a891d4b-f123-412f-8810-ae355a07ffbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:16.886662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:16.886639    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:16.888088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:16.888058    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-xks-ca" pod "3a891d4b-f123-412f-8810-ae355a07ffbd" (UID: "3a891d4b-f123-412f-8810-ae355a07ffbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:17:16.888232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:16.888117    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a891d4b-f123-412f-8810-ae355a07ffbd-kube-api-access-7k4mp" pod "3a891d4b-f123-412f-8810-ae355a07ffbd" (UID: "3a891d4b-f123-412f-8810-ae355a07ffbd"). InnerVolumeSpecName "kube-api-access-7k4mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:16.988416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:16.988372    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k4mp\" (UniqueName: \"kubernetes.io/projected/3a891d4b-f123-412f-8810-ae355a07ffbd-kube-api-access-7k4mp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:16.988416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:16.988415    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a891d4b-f123-412f-8810-ae355a07ffbd-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:17.154668 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.154633    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"]
Nov 02 13:17:17.155770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.155631    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3a891d4b-f123-412f-8810-ae355a07ffbd" containerName="revoke-keys"
Nov 02 13:17:17.155770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.155655    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a891d4b-f123-412f-8810-ae355a07ffbd" containerName="revoke-keys"
Nov 02 13:17:17.156012 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.155863    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a891d4b-f123-412f-8810-ae355a07ffbd" containerName="revoke-keys"
Nov 02 13:17:17.159904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.159886    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.184398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.184362    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"]
Nov 02 13:17:17.187085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.187044    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-48d74e\"/\"kube-root-ca.crt\""
Nov 02 13:17:17.187241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.187158    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-48d74e\"/\"openshift-service-ca.crt\""
Nov 02 13:17:17.187241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.187172    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-48d74e\"/\"test-model-48d74e-kserve-self-signed-certs\""
Nov 02 13:17:17.187241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.187230    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-48d74e\"/\"default-dockercfg-l72vc\""
Nov 02 13:17:17.235060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.235028    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:17.294274 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.294235    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmjk\" (UniqueName: \"kubernetes.io/projected/e168fd9f-aa07-4650-b616-37ee7b92ee77-kube-api-access-kzmjk\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.294463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.294410    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-dshm\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.294463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.294430    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e168fd9f-aa07-4650-b616-37ee7b92ee77-tls-certs\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.294602 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.294575    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-tmp-dir\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.294673 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.294657    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-home\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.294728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.294693    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-model-cache\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.401027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.400053    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-dshm\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.401027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.400123    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e168fd9f-aa07-4650-b616-37ee7b92ee77-tls-certs\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.401027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.400243    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-tmp-dir\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.401027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.400309    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-home\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.401027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.400371    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-model-cache\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.401027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.400611    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmjk\" (UniqueName: \"kubernetes.io/projected/e168fd9f-aa07-4650-b616-37ee7b92ee77-kube-api-access-kzmjk\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.404070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.402351    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-tmp-dir\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.404070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.402920    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-home\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.404070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.403185    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-model-cache\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.412063 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.408893    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-dshm\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.414707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.414680    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e168fd9f-aa07-4650-b616-37ee7b92ee77-tls-certs\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.431014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.429377    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmjk\" (UniqueName: \"kubernetes.io/projected/e168fd9f-aa07-4650-b616-37ee7b92ee77-kube-api-access-kzmjk\") pod \"test-model-48d74e-kserve-85dcfc4f5f-rtsl2\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") " pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.480369 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.476308    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:17.677534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.677451    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"
Nov 02 13:17:17.677534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.677465    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j" event={"ID":"3a891d4b-f123-412f-8810-ae355a07ffbd","Type":"ContainerDied","Data":"9175456c8b856bbc2b760bf991c8bd3d00e2d8fd14b45b48049b39bdf656a7fe"}
Nov 02 13:17:17.677534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.677518    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9175456c8b856bbc2b760bf991c8bd3d00e2d8fd14b45b48049b39bdf656a7fe"
Nov 02 13:17:17.862919 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:17:17.862877    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode168fd9f_aa07_4650_b616_37ee7b92ee77.slice/crio-43fb440c066afeec1190d55c6d50a952a01d43139b3db13bf335239fadc33f7b WatchSource:0}: Error finding container 43fb440c066afeec1190d55c6d50a952a01d43139b3db13bf335239fadc33f7b: Status 404 returned error can't find the container with id 43fb440c066afeec1190d55c6d50a952a01d43139b3db13bf335239fadc33f7b
Nov 02 13:17:17.864889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:17.864865    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"]
Nov 02 13:17:18.545233 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:18.545194    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:18.688426 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:18.688374    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2" event={"ID":"e168fd9f-aa07-4650-b616-37ee7b92ee77","Type":"ContainerStarted","Data":"84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58"}
Nov 02 13:17:18.688638 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:18.688443    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2" event={"ID":"e168fd9f-aa07-4650-b616-37ee7b92ee77","Type":"ContainerStarted","Data":"43fb440c066afeec1190d55c6d50a952a01d43139b3db13bf335239fadc33f7b"}
Nov 02 13:17:18.737860 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:18.737804    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2" podStartSLOduration=1.737790404 podStartE2EDuration="1.737790404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:18.73455557 +0000 UTC m=+2984.841258413" watchObservedRunningTime="2026-11-02 13:17:18.737790404 +0000 UTC m=+2984.844493254"
Nov 02 13:17:19.477304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:19.477268    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:20.199163 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:20.199124    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:21.901747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:21.901712    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:23.561787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:23.561743    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:24.647749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:24.647721    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"
Nov 02 13:17:24.648168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:24.647788    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"
Nov 02 13:17:25.071649 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.071554    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:25.199090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.199044    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:25.401890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.401858    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:25.608985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.608951    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"]
Nov 02 13:17:25.609217 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.609192    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q" podUID="f1c7693b-d870-4917-afc2-c27e80fe2b69" containerName="sim" containerID="cri-o://cec7dbff10f8e1cc401c8e54723b26ac1a8f852d5ab32d09a0a9051c47a06c0e" gracePeriod=30
Nov 02 13:17:25.830869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.830830    5879 generic.go:356] "Generic (PLEG): container finished" podID="f1c7693b-d870-4917-afc2-c27e80fe2b69" containerID="cec7dbff10f8e1cc401c8e54723b26ac1a8f852d5ab32d09a0a9051c47a06c0e" exitCode=0
Nov 02 13:17:25.831434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.831023    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q" event={"ID":"f1c7693b-d870-4917-afc2-c27e80fe2b69","Type":"ContainerDied","Data":"cec7dbff10f8e1cc401c8e54723b26ac1a8f852d5ab32d09a0a9051c47a06c0e"}
Nov 02 13:17:25.845466 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.845349    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:25.923967 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.923727    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"]
Nov 02 13:17:25.931385 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.931355    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:25.935878 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.935843    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"]
Nov 02 13:17:25.951649 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.951392    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"openshift-service-ca.crt\""
Nov 02 13:17:25.959408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.959382    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"agc-cred-0c5fa1a8b1b9\""
Nov 02 13:17:25.959737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.959704    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"payload-processing-external-model-e2e--49869bd7-dockercfg-j49vg\""
Nov 02 13:17:25.959879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.959784    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"routing-overlay\""
Nov 02 13:17:25.959879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.959845    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"kube-root-ca.crt\""
Nov 02 13:17:25.960019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:25.959886    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-740f09\"/\"payload-processing-external-model-plugins-e2e-extmod-a-740f09\""
Nov 02 13:17:26.068792 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.068679    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.068792 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.068732    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.069050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.068949    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shtk7\" (UniqueName: \"kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-kube-api-access-shtk7\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.069195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.069176    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.120593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.120553    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"
Nov 02 13:17:26.176017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.170376    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.176017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.170576    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtk7\" (UniqueName: \"kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-kube-api-access-shtk7\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.176017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.172426    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.176017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.172559    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.176017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.173482    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.180016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.179591    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.272017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.258709    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.272017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.259493    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shtk7\" (UniqueName: \"kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-kube-api-access-shtk7\") pod \"payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") " pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.272017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.271830    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.275028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.273788    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c7693b-d870-4917-afc2-c27e80fe2b69-kube-api-access-xgd5k\" (UniqueName: \"kubernetes.io/projected/f1c7693b-d870-4917-afc2-c27e80fe2b69-kube-api-access-xgd5k\") pod \"f1c7693b-d870-4917-afc2-c27e80fe2b69\" (UID: \"f1c7693b-d870-4917-afc2-c27e80fe2b69\") "
Nov 02 13:17:26.275028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.274149    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1c7693b-d870-4917-afc2-c27e80fe2b69-tls\" (UniqueName: \"kubernetes.io/secret/f1c7693b-d870-4917-afc2-c27e80fe2b69-tls\") pod \"f1c7693b-d870-4917-afc2-c27e80fe2b69\" (UID: \"f1c7693b-d870-4917-afc2-c27e80fe2b69\") "
Nov 02 13:17:26.285082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.277865    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c7693b-d870-4917-afc2-c27e80fe2b69-kube-api-access-xgd5k" pod "f1c7693b-d870-4917-afc2-c27e80fe2b69" (UID: "f1c7693b-d870-4917-afc2-c27e80fe2b69"). InnerVolumeSpecName "kube-api-access-xgd5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:26.285082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.282106    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1c7693b-d870-4917-afc2-c27e80fe2b69-tls" pod "f1c7693b-d870-4917-afc2-c27e80fe2b69" (UID: "f1c7693b-d870-4917-afc2-c27e80fe2b69"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:17:26.376929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.376851    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgd5k\" (UniqueName: \"kubernetes.io/projected/f1c7693b-d870-4917-afc2-c27e80fe2b69-kube-api-access-xgd5k\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:26.376929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.376905    5879 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f1c7693b-d870-4917-afc2-c27e80fe2b69-tls\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:26.610521 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:17:26.610448    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be2956d_1924_4cc0_9419_291c525d1827.slice/crio-7287d199cc5b57519df382c94ede4e1fad3ab41c903d6cc2ee7550b2354a4785 WatchSource:0}: Error finding container 7287d199cc5b57519df382c94ede4e1fad3ab41c903d6cc2ee7550b2354a4785: Status 404 returned error can't find the container with id 7287d199cc5b57519df382c94ede4e1fad3ab41c903d6cc2ee7550b2354a4785
Nov 02 13:17:26.612137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.612074    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"]
Nov 02 13:17:26.843688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.843639    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z" event={"ID":"6be2956d-1924-4cc0-9419-291c525d1827","Type":"ContainerStarted","Data":"028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933"}
Nov 02 13:17:26.844228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.843706    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z" event={"ID":"6be2956d-1924-4cc0-9419-291c525d1827","Type":"ContainerStarted","Data":"7287d199cc5b57519df382c94ede4e1fad3ab41c903d6cc2ee7550b2354a4785"}
Nov 02 13:17:26.844228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.843796    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:26.845554 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.845530    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"
Nov 02 13:17:26.845554 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.845554    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q" event={"ID":"f1c7693b-d870-4917-afc2-c27e80fe2b69","Type":"ContainerDied","Data":"e8bb7d90adc8c0c7b2d24048c69e88bb47cdc3d4dd86d811aea872e92e0afad4"}
Nov 02 13:17:26.845763 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.845595    5879 scope.go:122] "RemoveContainer" containerID="cec7dbff10f8e1cc401c8e54723b26ac1a8f852d5ab32d09a0a9051c47a06c0e"
Nov 02 13:17:26.930581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:26.930526    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z" podStartSLOduration=1.930512182 podStartE2EDuration="1.930512182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:26.926280747 +0000 UTC m=+2993.032983598" watchObservedRunningTime="2026-11-02 13:17:26.930512182 +0000 UTC m=+2993.037215032"
Nov 02 13:17:27.020384 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.020336    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"]
Nov 02 13:17:27.024778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.024749    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-6a71db/e2e-usage-sim-233c03-8c55d4647-bqz8q"]
Nov 02 13:17:27.062263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.062223    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"]
Nov 02 13:17:27.064038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.063404    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f1c7693b-d870-4917-afc2-c27e80fe2b69" containerName="sim"
Nov 02 13:17:27.064038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.063432    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c7693b-d870-4917-afc2-c27e80fe2b69" containerName="sim"
Nov 02 13:17:27.064038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.063674    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c7693b-d870-4917-afc2-c27e80fe2b69" containerName="sim"
Nov 02 13:17:27.070140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.070109    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.097199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.097166    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"]
Nov 02 13:17:27.108134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.108087    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"agc-cred-148dac49c765\""
Nov 02 13:17:27.108389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.108366    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"payload-processing-external-model-e2e--4b89980e-dockercfg-lw85f\""
Nov 02 13:17:27.109138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.108880    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"payload-processing-external-model-plugins-e2e-extmod-b-90cda9\""
Nov 02 13:17:27.109138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.108954    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"openshift-service-ca.crt\""
Nov 02 13:17:27.109138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.108982    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"kube-root-ca.crt\""
Nov 02 13:17:27.109138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.109103    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-90cda9\"/\"routing-overlay\""
Nov 02 13:17:27.190614 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.190525    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.190820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.190798    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh5fh\" (UniqueName: \"kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-kube-api-access-xh5fh\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.190879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.190853    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.191242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.190931    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.218403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.218369    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:27.292564 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.292507    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.292770 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:27.292694    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-90cda9/agc-cred-148dac49c765: secret "agc-cred-148dac49c765" not found
Nov 02 13:17:27.292770 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:27.292721    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62: secret "agc-cred-148dac49c765" not found
Nov 02 13:17:27.292770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.292747    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh5fh\" (UniqueName: \"kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-kube-api-access-xh5fh\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.292956 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:27.292806    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials podName:ce9e90d6-4928-4d40-85cb-48a690478171 nodeName:}" failed. No retries permitted until 2026-11-02 13:17:27.792781046 +0000 UTC m=+2993.899483895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62" (UID: "ce9e90d6-4928-4d40-85cb-48a690478171") : secret "agc-cred-148dac49c765" not found
Nov 02 13:17:27.292956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.292829    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.292956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.292887    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.293583 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.293558    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.293838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.293815    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.321722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.321688    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh5fh\" (UniqueName: \"kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-kube-api-access-xh5fh\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.343645 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.343619    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:27.480073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.478419    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:27.511068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.511027    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:27.813279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.813184    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:27.813513 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:27.813493    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-90cda9/agc-cred-148dac49c765: secret "agc-cred-148dac49c765" not found
Nov 02 13:17:27.813579 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:27.813517    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62: secret "agc-cred-148dac49c765" not found
Nov 02 13:17:27.813636 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:27.813623    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials podName:ce9e90d6-4928-4d40-85cb-48a690478171 nodeName:}" failed. No retries permitted until 2026-11-02 13:17:28.813569622 +0000 UTC m=+2994.920272466 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62" (UID: "ce9e90d6-4928-4d40-85cb-48a690478171") : secret "agc-cred-148dac49c765" not found
Nov 02 13:17:27.867962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.867934    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:17:27.885980 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:27.885949    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:28.017107 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:28.017073    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-740f09/agc-cred-0c5fa1a8b1b9: secret "agc-cred-0c5fa1a8b1b9" not found
Nov 02 13:17:28.017107 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:28.017100    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z: secret "agc-cred-0c5fa1a8b1b9" not found
Nov 02 13:17:28.017338 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:28.017171    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials podName:6be2956d-1924-4cc0-9419-291c525d1827 nodeName:}" failed. No retries permitted until 2026-11-02 13:17:28.517151538 +0000 UTC m=+2994.623854378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z" (UID: "6be2956d-1924-4cc0-9419-291c525d1827") : secret "agc-cred-0c5fa1a8b1b9" not found
Nov 02 13:17:28.022833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:28.022249    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c7693b-d870-4917-afc2-c27e80fe2b69" path="/var/lib/kubelet/pods/f1c7693b-d870-4917-afc2-c27e80fe2b69/volumes"
Nov 02 13:17:28.443288 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:28.443256    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"]
Nov 02 13:17:28.524535 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:28.524504    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-740f09/agc-cred-0c5fa1a8b1b9: secret "agc-cred-0c5fa1a8b1b9" not found
Nov 02 13:17:28.524535 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:28.524529    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z: secret "agc-cred-0c5fa1a8b1b9" not found
Nov 02 13:17:28.524779 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:28.524589    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials podName:6be2956d-1924-4cc0-9419-291c525d1827 nodeName:}" failed. No retries permitted until 2026-11-02 13:17:29.52457234 +0000 UTC m=+2995.631275186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z" (UID: "6be2956d-1924-4cc0-9419-291c525d1827") : secret "agc-cred-0c5fa1a8b1b9" not found
Nov 02 13:17:28.550796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:28.550751    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:28.834880 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:28.830452    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:28.834880 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:28.830550    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-90cda9/agc-cred-148dac49c765: secret "agc-cred-148dac49c765" not found
Nov 02 13:17:28.834880 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:28.830586    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62: secret "agc-cred-148dac49c765" not found
Nov 02 13:17:28.834880 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:28.830672    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials podName:ce9e90d6-4928-4d40-85cb-48a690478171 nodeName:}" failed. No retries permitted until 2026-11-02 13:17:30.830644954 +0000 UTC m=+2996.937347787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62" (UID: "ce9e90d6-4928-4d40-85cb-48a690478171") : secret "agc-cred-148dac49c765" not found
Nov 02 13:17:29.072200 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:29.072163    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:29.478169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:29.478137    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"]
Nov 02 13:17:29.478579 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:29.478560    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62" podUID="ce9e90d6-4928-4d40-85cb-48a690478171"
Nov 02 13:17:29.543668 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:29.543630    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-740f09/agc-cred-0c5fa1a8b1b9: secret "agc-cred-0c5fa1a8b1b9" not found
Nov 02 13:17:29.543668 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:29.543659    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z: secret "agc-cred-0c5fa1a8b1b9" not found
Nov 02 13:17:29.543878 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:29.543724    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials podName:6be2956d-1924-4cc0-9419-291c525d1827 nodeName:}" failed. No retries permitted until 2026-11-02 13:17:31.543705539 +0000 UTC m=+2997.650408385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z" (UID: "6be2956d-1924-4cc0-9419-291c525d1827") : secret "agc-cred-0c5fa1a8b1b9" not found
Nov 02 13:17:29.589302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:29.589263    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:29.930903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:29.930861    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:29.950019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:29.945133    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:30.051036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.050969    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-plugins-config-volume\") pod \"ce9e90d6-4928-4d40-85cb-48a690478171\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") "
Nov 02 13:17:30.051227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.051092    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-routing-overlay-volume\") pod \"ce9e90d6-4928-4d40-85cb-48a690478171\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") "
Nov 02 13:17:30.051278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.051226    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-kube-api-access-xh5fh\" (UniqueName: \"kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-kube-api-access-xh5fh\") pod \"ce9e90d6-4928-4d40-85cb-48a690478171\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") "
Nov 02 13:17:30.051433 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.051407    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-plugins-config-volume" pod "ce9e90d6-4928-4d40-85cb-48a690478171" (UID: "ce9e90d6-4928-4d40-85cb-48a690478171"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:30.051504 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.051481    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-routing-overlay-volume" pod "ce9e90d6-4928-4d40-85cb-48a690478171" (UID: "ce9e90d6-4928-4d40-85cb-48a690478171"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:30.052144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.052118    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:30.052144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.052147    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce9e90d6-4928-4d40-85cb-48a690478171-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:30.053643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.053611    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-kube-api-access-xh5fh" pod "ce9e90d6-4928-4d40-85cb-48a690478171" (UID: "ce9e90d6-4928-4d40-85cb-48a690478171"). InnerVolumeSpecName "kube-api-access-xh5fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:30.154136 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.154095    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh5fh\" (UniqueName: \"kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-kube-api-access-xh5fh\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:30.297477 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.297390    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"]
Nov 02 13:17:30.302655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.302630    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.328784 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.328743    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"]
Nov 02 13:17:30.400176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.400140    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6563ed-openshift-default-dockercfg-dh4qv\""
Nov 02 13:17:30.459584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.459552    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.459802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.459602    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.459802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.459759    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f561b255-e7a4-4234-981e-de48e987b6f7-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.459802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.459788    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.459950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.459826    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.459950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.459900    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45p82\" (UniqueName: \"kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-kube-api-access-45p82\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.459950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.459930    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.460081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.459968    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f561b255-e7a4-4234-981e-de48e987b6f7-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.460167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.460137    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.562537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.562443    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.562537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.562506    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.562764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.562588    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f561b255-e7a4-4234-981e-de48e987b6f7-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.562764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.562630    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.562764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.562677    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.562764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.562730    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45p82\" (UniqueName: \"kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-kube-api-access-45p82\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.563040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.562839    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.563040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.562874    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.563040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.562940    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f561b255-e7a4-4234-981e-de48e987b6f7-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.563227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.563041    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.563227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.563178    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.563227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.563185    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.563411 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.563389    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.563679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.563656    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f561b255-e7a4-4234-981e-de48e987b6f7-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.565442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.565419    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.565557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.565540    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f561b255-e7a4-4234-981e-de48e987b6f7-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.588282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.588254    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.588421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.588402    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45p82\" (UniqueName: \"kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-kube-api-access-45p82\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.620802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.620772    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:30.750553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.750522    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:30.873848 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.873815    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62\" (UID: \"ce9e90d6-4928-4d40-85cb-48a690478171\") " pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:30.874063 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:30.873976    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-90cda9/agc-cred-148dac49c765: secret "agc-cred-148dac49c765" not found
Nov 02 13:17:30.874063 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:30.874033    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62: secret "agc-cred-148dac49c765" not found
Nov 02 13:17:30.874165 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:30.874089    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials podName:ce9e90d6-4928-4d40-85cb-48a690478171 nodeName:}" failed. No retries permitted until 2026-11-02 13:17:34.874072393 +0000 UTC m=+3000.980775223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62" (UID: "ce9e90d6-4928-4d40-85cb-48a690478171") : secret "agc-cred-148dac49c765" not found
Nov 02 13:17:30.946512 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.946482    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"
Nov 02 13:17:30.948694 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.948667    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"]
Nov 02 13:17:30.949083 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:17:30.949055    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf561b255_e7a4_4234_981e_de48e987b6f7.slice/crio-1da9bb2a8c1aea2275c4658ba6c4064d032087ae8ae2607de01495d3482727be WatchSource:0}: Error finding container 1da9bb2a8c1aea2275c4658ba6c4064d032087ae8ae2607de01495d3482727be: Status 404 returned error can't find the container with id 1da9bb2a8c1aea2275c4658ba6c4064d032087ae8ae2607de01495d3482727be
Nov 02 13:17:30.952911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.952875    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:17:30.953090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.952966    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:17:30.953090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:30.953025    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:17:31.034441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:31.034404    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"]
Nov 02 13:17:31.038865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:31.038835    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-90cda9/payload-processing-external-model-e2e-extmod-b-90cda9-5d6cg9q62"]
Nov 02 13:17:31.180137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:31.180043    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce9e90d6-4928-4d40-85cb-48a690478171-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:31.587181 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:31.587080    5879 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-740f09/agc-cred-0c5fa1a8b1b9: secret "agc-cred-0c5fa1a8b1b9" not found
Nov 02 13:17:31.587181 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:31.587118    5879 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z: secret "agc-cred-0c5fa1a8b1b9" not found
Nov 02 13:17:31.587405 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:31.587194    5879 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials podName:6be2956d-1924-4cc0-9419-291c525d1827 nodeName:}" failed. No retries permitted until 2026-11-02 13:17:35.587176063 +0000 UTC m=+3001.693878911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z" (UID: "6be2956d-1924-4cc0-9419-291c525d1827") : secret "agc-cred-0c5fa1a8b1b9" not found
Nov 02 13:17:31.957306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:31.957262    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj" event={"ID":"f561b255-e7a4-4234-981e-de48e987b6f7","Type":"ContainerStarted","Data":"002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628"}
Nov 02 13:17:31.957535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:31.957320    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj" event={"ID":"f561b255-e7a4-4234-981e-de48e987b6f7","Type":"ContainerStarted","Data":"1da9bb2a8c1aea2275c4658ba6c4064d032087ae8ae2607de01495d3482727be"}
Nov 02 13:17:32.008059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:32.007694    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj" podStartSLOduration=2.007672595 podStartE2EDuration="2.007672595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:32.006524267 +0000 UTC m=+2998.113227130" watchObservedRunningTime="2026-11-02 13:17:32.007672595 +0000 UTC m=+2998.114375446"
Nov 02 13:17:32.026847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:32.026810    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce9e90d6-4928-4d40-85cb-48a690478171" path="/var/lib/kubelet/pods/ce9e90d6-4928-4d40-85cb-48a690478171/volumes"
Nov 02 13:17:32.538742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:32.538703    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:32.621243 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:32.621201    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:32.623496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:32.623466    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body=
Nov 02 13:17:32.623637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:32.623518    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj" podUID="f561b255-e7a4-4234-981e-de48e987b6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused"
Nov 02 13:17:33.621631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:33.621588    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body=
Nov 02 13:17:33.622073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:33.621658    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj" podUID="f561b255-e7a4-4234-981e-de48e987b6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused"
Nov 02 13:17:33.911079 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:33.902609    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z" podUID="6be2956d-1924-4cc0-9419-291c525d1827" containerName="payload-processing" containerID="cri-o://028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933" gracePeriod=25
Nov 02 13:17:34.256952 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.256908    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:34.263690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.263660    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:34.350296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.350096    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-kube-api-access-shtk7\" (UniqueName: \"kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-kube-api-access-shtk7\") pod \"6be2956d-1924-4cc0-9419-291c525d1827\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") "
Nov 02 13:17:34.350296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.350269    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-routing-overlay-volume\") pod \"6be2956d-1924-4cc0-9419-291c525d1827\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") "
Nov 02 13:17:34.350757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.350364    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-plugins-config-volume\") pod \"6be2956d-1924-4cc0-9419-291c525d1827\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") "
Nov 02 13:17:34.350757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.350494    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials\") pod \"6be2956d-1924-4cc0-9419-291c525d1827\" (UID: \"6be2956d-1924-4cc0-9419-291c525d1827\") "
Nov 02 13:17:34.350891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.350820    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-routing-overlay-volume" pod "6be2956d-1924-4cc0-9419-291c525d1827" (UID: "6be2956d-1924-4cc0-9419-291c525d1827"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:34.351253 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.351224    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-plugins-config-volume" pod "6be2956d-1924-4cc0-9419-291c525d1827" (UID: "6be2956d-1924-4cc0-9419-291c525d1827"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:17:34.351376 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.351290    5879 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-routing-overlay-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:34.356044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.353919    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials" pod "6be2956d-1924-4cc0-9419-291c525d1827" (UID: "6be2956d-1924-4cc0-9419-291c525d1827"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:34.358317 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.358283    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-kube-api-access-shtk7" pod "6be2956d-1924-4cc0-9419-291c525d1827" (UID: "6be2956d-1924-4cc0-9419-291c525d1827"). InnerVolumeSpecName "kube-api-access-shtk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:17:34.452582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.452547    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be2956d-1924-4cc0-9419-291c525d1827-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:34.452582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.452576    5879 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-provider-credentials\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:34.452582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.452586    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shtk7\" (UniqueName: \"kubernetes.io/projected/6be2956d-1924-4cc0-9419-291c525d1827-kube-api-access-shtk7\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:17:34.512055 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.511949    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:34.621202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.621169    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body=
Nov 02 13:17:34.621418 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.621239    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj" podUID="f561b255-e7a4-4234-981e-de48e987b6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused"
Nov 02 13:17:34.658136 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.658100    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:34.938939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:34.938905    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:35.008535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.008501    5879 generic.go:356] "Generic (PLEG): container finished" podID="6be2956d-1924-4cc0-9419-291c525d1827" containerID="028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933" exitCode=0
Nov 02 13:17:35.008535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.008541    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"
Nov 02 13:17:35.008832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.008590    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z" event={"ID":"6be2956d-1924-4cc0-9419-291c525d1827","Type":"ContainerDied","Data":"028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933"}
Nov 02 13:17:35.008832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.008707    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z" event={"ID":"6be2956d-1924-4cc0-9419-291c525d1827","Type":"ContainerDied","Data":"7287d199cc5b57519df382c94ede4e1fad3ab41c903d6cc2ee7550b2354a4785"}
Nov 02 13:17:35.008832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.008718    5879 scope.go:122] "RemoveContainer" containerID="028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933"
Nov 02 13:17:35.021024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.020985    5879 scope.go:122] "RemoveContainer" containerID="028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933"
Nov 02 13:17:35.021364 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:35.021333    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933\": container with ID starting with 028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933 not found: ID does not exist" containerID="028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933"
Nov 02 13:17:35.021413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.021376    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933"} err="failed to get container status \"028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933\": rpc error: code = NotFound desc = could not find container \"028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933\": container with ID starting with 028709714b6fb48972443393df5c9a502359e2cf2c1b972787d1d4c6adddc933 not found: ID does not exist"
Nov 02 13:17:35.069139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.069096    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"]
Nov 02 13:17:35.075339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.075309    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-740f09/payload-processing-external-model-e2e-extmod-a-740f09-79fb9q58z"]
Nov 02 13:17:35.420249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.420207    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:35.601015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.598829    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:35.626434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.626400    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body=
Nov 02 13:17:35.628356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:35.628317    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj" podUID="f561b255-e7a4-4234-981e-de48e987b6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused"
Nov 02 13:17:36.023941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:36.023901    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be2956d-1924-4cc0-9419-291c525d1827" path="/var/lib/kubelet/pods/6be2956d-1924-4cc0-9419-291c525d1827/volumes"
Nov 02 13:17:36.420396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:36.420361    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:36.621929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:36.621895    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body=
Nov 02 13:17:36.622243 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:36.621966    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj" podUID="f561b255-e7a4-4234-981e-de48e987b6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused"
Nov 02 13:17:37.172523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:37.172235    5879 scope.go:122] "RemoveContainer" containerID="d3a945eb645adb146fb96fd1fbcdc4497c4d4290a1c44d12c16020c39f6ad943"
Nov 02 13:17:37.621904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:37.621866    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body=
Nov 02 13:17:37.622137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:37.621935    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj" podUID="f561b255-e7a4-4234-981e-de48e987b6f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused"
Nov 02 13:17:38.119662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:38.119628    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:38.626987 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:38.626955    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:38.627474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:38.627456    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:38.628315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:38.628292    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:17:39.455307 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:39.455269    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:39.599881 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:39.599850    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:39.980752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:39.978899    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:41.148488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.148451    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:41.290633 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.290594    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"]
Nov 02 13:17:41.291018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.290976    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:41.291679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.291657    5879 cpu_manager.go:414] "containerMap: removing container" podUID="6be2956d-1924-4cc0-9419-291c525d1827" containerName="payload-processing"
Nov 02 13:17:41.291679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.291676    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be2956d-1924-4cc0-9419-291c525d1827" containerName="payload-processing"
Nov 02 13:17:41.291855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.291804    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be2956d-1924-4cc0-9419-291c525d1827" containerName="payload-processing"
Nov 02 13:17:41.297564 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.297540    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"
Nov 02 13:17:41.319680 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.319648    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"]
Nov 02 13:17:41.327323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.327055    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-23f720\""
Nov 02 13:17:41.327323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.327260    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-23f720-dockercfg-k58fx\""
Nov 02 13:17:41.408069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.402142    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"]
Nov 02 13:17:41.414017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.408835    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"
Nov 02 13:17:41.443090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.434905    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhxf\" (UniqueName: \"kubernetes.io/projected/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-kube-api-access-gvhxf\") pod \"payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g\" (UID: \"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"
Nov 02 13:17:41.443090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.435050    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g\" (UID: \"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"
Nov 02 13:17:41.521889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.521762    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"]
Nov 02 13:17:41.538474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.538317    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbfx9\" (UniqueName: \"kubernetes.io/projected/fdee021c-70d3-47c0-be6b-589d619ce6bc-kube-api-access-mbfx9\") pod \"payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4\" (UID: \"fdee021c-70d3-47c0-be6b-589d619ce6bc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"
Nov 02 13:17:41.538841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.538809    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g\" (UID: \"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"
Nov 02 13:17:41.539327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.539100    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdee021c-70d3-47c0-be6b-589d619ce6bc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4\" (UID: \"fdee021c-70d3-47c0-be6b-589d619ce6bc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"
Nov 02 13:17:41.539892 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.539854    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhxf\" (UniqueName: \"kubernetes.io/projected/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-kube-api-access-gvhxf\") pod \"payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g\" (UID: \"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"
Nov 02 13:17:41.542020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.541968    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g\" (UID: \"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"
Nov 02 13:17:41.569787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.569756    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhxf\" (UniqueName: \"kubernetes.io/projected/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-kube-api-access-gvhxf\") pod \"payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g\" (UID: \"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"
Nov 02 13:17:41.593643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.593532    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:41.615060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.614598    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"
Nov 02 13:17:41.641867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.641839    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbfx9\" (UniqueName: \"kubernetes.io/projected/fdee021c-70d3-47c0-be6b-589d619ce6bc-kube-api-access-mbfx9\") pod \"payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4\" (UID: \"fdee021c-70d3-47c0-be6b-589d619ce6bc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"
Nov 02 13:17:41.642041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.641903    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdee021c-70d3-47c0-be6b-589d619ce6bc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4\" (UID: \"fdee021c-70d3-47c0-be6b-589d619ce6bc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"
Nov 02 13:17:41.642774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.642747    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdee021c-70d3-47c0-be6b-589d619ce6bc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4\" (UID: \"fdee021c-70d3-47c0-be6b-589d619ce6bc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"
Nov 02 13:17:41.669090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.669018    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbfx9\" (UniqueName: \"kubernetes.io/projected/fdee021c-70d3-47c0-be6b-589d619ce6bc-kube-api-access-mbfx9\") pod \"payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4\" (UID: \"fdee021c-70d3-47c0-be6b-589d619ce6bc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"
Nov 02 13:17:41.821663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.821631    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"
Nov 02 13:17:41.953093 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:41.953061    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"]
Nov 02 13:17:41.958200 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:17:41.958153    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22afe41_b0e8_4ef4_8ce2_50d5ba0d4c1f.slice/crio-4a380114ab41b7e7eb58993dfeb849636a3448d6f2645cf8c676bd455f8cf987 WatchSource:0}: Error finding container 4a380114ab41b7e7eb58993dfeb849636a3448d6f2645cf8c676bd455f8cf987: Status 404 returned error can't find the container with id 4a380114ab41b7e7eb58993dfeb849636a3448d6f2645cf8c676bd455f8cf987
Nov 02 13:17:42.078655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:42.078596    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g" event={"ID":"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f","Type":"ContainerStarted","Data":"4a380114ab41b7e7eb58993dfeb849636a3448d6f2645cf8c676bd455f8cf987"}
Nov 02 13:17:42.139698 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:17:42.139668    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdee021c_70d3_47c0_be6b_589d619ce6bc.slice/crio-afbe09d8b17be228d8d1fc313768975a26ce15d5272c22f33ce730f5f82525aa WatchSource:0}: Error finding container afbe09d8b17be228d8d1fc313768975a26ce15d5272c22f33ce730f5f82525aa: Status 404 returned error can't find the container with id afbe09d8b17be228d8d1fc313768975a26ce15d5272c22f33ce730f5f82525aa
Nov 02 13:17:42.141805 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:42.141782    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"]
Nov 02 13:17:43.089196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:43.089160    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4" event={"ID":"fdee021c-70d3-47c0-be6b-589d619ce6bc","Type":"ContainerStarted","Data":"34f8909bbee0bd13282f52151111daebc2f49cc5e01c50b37aef31c83fea00af"}
Nov 02 13:17:43.089701 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:43.089216    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4" event={"ID":"fdee021c-70d3-47c0-be6b-589d619ce6bc","Type":"ContainerStarted","Data":"afbe09d8b17be228d8d1fc313768975a26ce15d5272c22f33ce730f5f82525aa"}
Nov 02 13:17:43.089701 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:43.089251    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"
Nov 02 13:17:43.090880 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:43.090854    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g" event={"ID":"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f","Type":"ContainerStarted","Data":"d6a5cb8977190f57036901d80f5e397c91bccbf2df77ed89a3aaf3d65eb9388b"}
Nov 02 13:17:43.091037 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:43.091026    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"
Nov 02 13:17:43.138459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:43.138398    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4" podStartSLOduration=2.138384118 podStartE2EDuration="2.138384118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:43.137589516 +0000 UTC m=+3009.244292367" watchObservedRunningTime="2026-11-02 13:17:43.138384118 +0000 UTC m=+3009.245086965"
Nov 02 13:17:43.183553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:43.183490    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g" podStartSLOduration=2.18347592 podStartE2EDuration="2.18347592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:17:43.181415184 +0000 UTC m=+3009.288118036" watchObservedRunningTime="2026-11-02 13:17:43.18347592 +0000 UTC m=+3009.290178771"
Nov 02 13:17:43.203630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:43.203596    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:44.823027 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:17:44.822969    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:17:44.957714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:44.957673    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:46.544427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:46.544390    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:48.034465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:48.034432    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"]
Nov 02 13:17:48.047688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:48.047656    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a6a05a-87wvj"]
Nov 02 13:17:48.128709 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:48.128670    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:49.034004 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:49.033968    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"]
Nov 02 13:17:49.048112 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:49.048076    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"]
Nov 02 13:17:49.052039 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:49.052020    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64975c55-7qh8l"]
Nov 02 13:17:49.055931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:49.055905    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdec91b2-pchqc"]
Nov 02 13:17:49.769579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:49.769543    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:50.020202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:50.020125    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a91b416-95e1-45b3-ae59-e713edf8c562" path="/var/lib/kubelet/pods/3a91b416-95e1-45b3-ae59-e713edf8c562/volumes"
Nov 02 13:17:50.020676 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:50.020662    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa1c331-dd6a-42d7-b210-5b50e9669626" path="/var/lib/kubelet/pods/5fa1c331-dd6a-42d7-b210-5b50e9669626/volumes"
Nov 02 13:17:50.021191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:50.021175    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3358d95-603a-4760-a73b-a28daf972475" path="/var/lib/kubelet/pods/c3358d95-603a-4760-a73b-a28daf972475/volumes"
Nov 02 13:17:51.444333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:51.444295    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:52.967097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:52.967063    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:54.102895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:54.102860    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"
Nov 02 13:17:54.103326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:54.102943    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"
Nov 02 13:17:54.658635 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:54.658599    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:56.285223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:56.285114    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:57.789113 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:57.789079    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:17:59.359559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:17:59.359525    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:00.040857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:00.040821    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"]
Nov 02 13:18:00.054592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:00.054558    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"]
Nov 02 13:18:00.058656 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:00.058626    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82853549-d7zk2"]
Nov 02 13:18:00.062302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:00.062267    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08754800-wl8gj"]
Nov 02 13:18:01.122024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:01.121867    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:01.253928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:01.253895    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:01.583129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:01.583033    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:02.020309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:02.020271    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577acb2a-8bfc-4443-8e71-600fc40e5d1b" path="/var/lib/kubelet/pods/577acb2a-8bfc-4443-8e71-600fc40e5d1b/volumes"
Nov 02 13:18:02.020826 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:02.020812    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb1da1a-2eb8-4d89-a8f6-11d75673270b" path="/var/lib/kubelet/pods/6eb1da1a-2eb8-4d89-a8f6-11d75673270b/volumes"
Nov 02 13:18:03.039257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:03.039198    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:04.715167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:04.715128    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:06.326149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:06.326114    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:07.852165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:07.852130    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:09.480675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:09.480644    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:11.163236 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:11.163200    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:12.710554 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:12.710523    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:14.419469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:14.419436    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:14.823763 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:18:14.823671    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:18:15.457519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.457475    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"]
Nov 02 13:18:15.465046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.464964    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.488291 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.488257    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.488828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.488798    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.488984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.488930    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.488984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.488957    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p22cl\" (UniqueName: \"kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-kube-api-access-p22cl\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.489179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.489023    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.489179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.489050    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf97cd15-75c5-4b58-90b0-396865600faf-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.490414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.489183    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.490414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.489224    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf97cd15-75c5-4b58-90b0-396865600faf-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.490414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.489248    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.513659 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.513623    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"]
Nov 02 13:18:15.590365 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.590330    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.590571 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.590461    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.590571 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.590491    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p22cl\" (UniqueName: \"kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-kube-api-access-p22cl\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.590571 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.590521    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.590571 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.590538    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf97cd15-75c5-4b58-90b0-396865600faf-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.590816 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.590654    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.590816 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.590694    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf97cd15-75c5-4b58-90b0-396865600faf-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.590816 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.590719    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.590816 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.590749    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.591075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.590759    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.591075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.590857    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.591201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.591132    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.591201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.591156    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.591338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.591319    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf97cd15-75c5-4b58-90b0-396865600faf-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.593648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.593585    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf97cd15-75c5-4b58-90b0-396865600faf-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.593791 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.593756    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.617966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.617933    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p22cl\" (UniqueName: \"kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-kube-api-access-p22cl\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.618168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.617976    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:15.794413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:15.794327    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:16.081725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:16.081589    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:16.147434 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:18:16.147397    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf97cd15_75c5_4b58_90b0_396865600faf.slice/crio-72982c31f9f684fb45fd2e3b716a0d7abdabcf179eff48bcf06110830903f461 WatchSource:0}: Error finding container 72982c31f9f684fb45fd2e3b716a0d7abdabcf179eff48bcf06110830903f461: Status 404 returned error can't find the container with id 72982c31f9f684fb45fd2e3b716a0d7abdabcf179eff48bcf06110830903f461
Nov 02 13:18:16.149356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:16.149332    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"]
Nov 02 13:18:16.150687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:16.150651    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:18:16.150753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:16.150723    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:18:16.150792 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:16.150755    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:18:16.475549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:16.475510    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx" event={"ID":"cf97cd15-75c5-4b58-90b0-396865600faf","Type":"ContainerStarted","Data":"c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9"}
Nov 02 13:18:16.475987 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:16.475566    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx" event={"ID":"cf97cd15-75c5-4b58-90b0-396865600faf","Type":"ContainerStarted","Data":"72982c31f9f684fb45fd2e3b716a0d7abdabcf179eff48bcf06110830903f461"}
Nov 02 13:18:16.548963 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:16.548902    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx" podStartSLOduration=1.548881131 podStartE2EDuration="1.548881131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:18:16.545781524 +0000 UTC m=+3042.652484376" watchObservedRunningTime="2026-11-02 13:18:16.548881131 +0000 UTC m=+3042.655583982"
Nov 02 13:18:16.795498 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:16.795410    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:17.807018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:17.806289    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body=
Nov 02 13:18:17.807018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:17.806350    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx" podUID="cf97cd15-75c5-4b58-90b0-396865600faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused"
Nov 02 13:18:17.849167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:17.849137    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:18.798086 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:18.798046    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body=
Nov 02 13:18:18.798416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:18.798385    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx" podUID="cf97cd15-75c5-4b58-90b0-396865600faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused"
Nov 02 13:18:19.511638 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:19.511592    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:19.795562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:19.795461    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body=
Nov 02 13:18:19.796594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:19.796558    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx" podUID="cf97cd15-75c5-4b58-90b0-396865600faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused"
Nov 02 13:18:20.798019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:20.797454    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body=
Nov 02 13:18:20.798019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:20.797539    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx" podUID="cf97cd15-75c5-4b58-90b0-396865600faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused"
Nov 02 13:18:21.199240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:21.199175    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:21.806232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:21.806193    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body=
Nov 02 13:18:21.806766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:21.806255    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx" podUID="cf97cd15-75c5-4b58-90b0-396865600faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused"
Nov 02 13:18:22.795023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:22.794960    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body=
Nov 02 13:18:22.795226 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:22.795069    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx" podUID="cf97cd15-75c5-4b58-90b0-396865600faf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused"
Nov 02 13:18:22.855095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:22.855055    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:23.800431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:23.800405    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:23.800695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:23.800671    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:23.801391 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:23.801372    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:18:24.638272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:24.638234    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:26.156139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:26.156105    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:27.850193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:27.850161    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:29.384951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:29.384916    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:30.490016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.489960    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"]
Nov 02 13:18:30.495838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.495812    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.521165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.521122    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"]
Nov 02 13:18:30.584854 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.584816    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.585074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.584869    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.585074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.584943    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.585074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.585007    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.585220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.585117    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg6db\" (UniqueName: \"kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-kube-api-access-zg6db\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.585264 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.585254    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.585387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.585374    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.585428 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.585409    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.585470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.585434    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.686720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.686683    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.686923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.686830    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.686923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.686878    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.686923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.686911    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.687129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.686975    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.687129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.687040    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.687129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.687084    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.687314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.687130    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.687314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.687180    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg6db\" (UniqueName: \"kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-kube-api-access-zg6db\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.687426 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.687339    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-credential-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.687426 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.687393    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-socket\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.687770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.687739    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-certs\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.687943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.687922    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-data\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.688140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.688116    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istiod-ca-cert\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.689324 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.689302    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-envoy\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.689945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.689927    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-podinfo\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.714827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.714801    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-token\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.714972 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.714948    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg6db\" (UniqueName: \"kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-kube-api-access-zg6db\") pod \"e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") " pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:30.813450 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:30.813360    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:31.031499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.031449    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:31.158502 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:18:31.158473    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f8c18c_bc7b_4475_a7b8_153e0b7648a8.slice/crio-62ee533cf5f1c48bb212f20b79bff84351b02cd59c0637686d01882285e41954 WatchSource:0}: Error finding container 62ee533cf5f1c48bb212f20b79bff84351b02cd59c0637686d01882285e41954: Status 404 returned error can't find the container with id 62ee533cf5f1c48bb212f20b79bff84351b02cd59c0637686d01882285e41954
Nov 02 13:18:31.160805 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.160775    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"]
Nov 02 13:18:31.161735 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.161699    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:18:31.161839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.161767    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:18:31.161839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.161806    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:18:31.170192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.170165    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:31.344342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.344311    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:31.551082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.550767    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:31.672266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.672227    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"]
Nov 02 13:18:31.672859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.672824    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh" podUID="8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" containerName="istio-proxy" containerID="cri-o://689eca75a6c64c18fe151f7e0f4dc4792947d20b537acffdd7a76d8ea365708a" gracePeriod=30
Nov 02 13:18:31.676884 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.676797    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz" event={"ID":"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8","Type":"ContainerStarted","Data":"f7ae730d323a87589c9bb46160a4bdbbfb55be057ff137489c7b8cbf47800744"}
Nov 02 13:18:31.677096 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.676923    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz" event={"ID":"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8","Type":"ContainerStarted","Data":"62ee533cf5f1c48bb212f20b79bff84351b02cd59c0637686d01882285e41954"}
Nov 02 13:18:31.803860 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.803747    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz" podStartSLOduration=1.803725987 podStartE2EDuration="1.803725987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:18:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:18:31.788426104 +0000 UTC m=+3057.895128977" watchObservedRunningTime="2026-11-02 13:18:31.803725987 +0000 UTC m=+3057.910428841"
Nov 02 13:18:31.819090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:31.818583    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:32.818019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:32.817747    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body=
Nov 02 13:18:32.818019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:32.817835    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz" podUID="d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused"
Nov 02 13:18:32.974174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:32.974139    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:33.813790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:33.813758    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body=
Nov 02 13:18:33.814012 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:33.813808    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz" podUID="d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused"
Nov 02 13:18:34.240261 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:34.240206    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"]
Nov 02 13:18:34.240795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:34.240417    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" podUID="5115dd72-ef9a-4a46-b84a-ee44b699082c" containerName="istio-proxy" containerID="cri-o://87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4" gracePeriod=30
Nov 02 13:18:34.813934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:34.813897    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body=
Nov 02 13:18:34.814211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:34.813973    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz" podUID="d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused"
Nov 02 13:18:35.030193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:35.030149    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:35.706302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:35.706260    5879 patch_prober.go:29] interesting pod/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:18:35.706938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:35.706887    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" podUID="5115dd72-ef9a-4a46-b84a-ee44b699082c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:18:35.813780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:35.813741    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body=
Nov 02 13:18:35.813976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:35.813795    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz" podUID="d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused"
Nov 02 13:18:36.816278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:36.816246    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body=
Nov 02 13:18:36.816924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:36.816892    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz" podUID="d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused"
Nov 02 13:18:36.838692 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:36.835087    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:37.066170 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.066143    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:18:37.128067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.128029    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-token\" (UniqueName: \"kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-token\") pod \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") "
Nov 02 13:18:37.128291 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.128201    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-data\") pod \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") "
Nov 02 13:18:37.128291 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.128261    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-podinfo\") pod \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") "
Nov 02 13:18:37.128458 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.128441    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istiod-ca-cert\") pod \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") "
Nov 02 13:18:37.128526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.128485    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-envoy\") pod \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") "
Nov 02 13:18:37.128588 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.128525    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-kube-api-access-xst8x\" (UniqueName: \"kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-kube-api-access-xst8x\") pod \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") "
Nov 02 13:18:37.128588 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.128522    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-data" pod "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" (UID: "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:37.128588 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.128571    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-socket\") pod \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") "
Nov 02 13:18:37.128835 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.128782    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istiod-ca-cert" pod "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" (UID: "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:18:37.128895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.128870    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-socket" pod "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" (UID: "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:37.128964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.128783    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-credential-socket\") pod \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") "
Nov 02 13:18:37.129074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.129019    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-credential-socket" pod "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" (UID: "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:37.129143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.129093    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-certs\") pod \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\" (UID: \"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7\") "
Nov 02 13:18:37.129419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.129389    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-certs" pod "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" (UID: "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:37.130927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.130570    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:37.130927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.130598    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:37.130927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.130615    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:37.130927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.130630    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:37.130927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.130644    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:37.131357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.131190    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-podinfo" pod "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" (UID: "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:18:37.133118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.133090    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-kube-api-access-xst8x" pod "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" (UID: "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7"). InnerVolumeSpecName "kube-api-access-xst8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:37.133338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.133314    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-envoy" pod "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" (UID: "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:37.133602 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.133582    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-token" pod "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" (UID: "8847cdca-ae49-47a9-8dcc-a7e655fc2ad7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:37.232045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.231829    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:37.232045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.231863    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:37.232045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.231873    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:37.232045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.231882    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xst8x\" (UniqueName: \"kubernetes.io/projected/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7-kube-api-access-xst8x\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:37.309561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.309473    5879 scope.go:122] "RemoveContainer" containerID="689eca75a6c64c18fe151f7e0f4dc4792947d20b537acffdd7a76d8ea365708a"
Nov 02 13:18:37.710742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.710707    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"]
Nov 02 13:18:37.710975 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.710951    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9" podUID="262814fe-6da4-48ae-85b8-c51aa9f2895a" containerName="istio-proxy" containerID="cri-o://82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941" gracePeriod=30
Nov 02 13:18:37.750239 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.750198    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"
Nov 02 13:18:37.760679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.756791    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh" event={"ID":"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7","Type":"ContainerDied","Data":"689eca75a6c64c18fe151f7e0f4dc4792947d20b537acffdd7a76d8ea365708a"}
Nov 02 13:18:37.760679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.756894    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh" event={"ID":"8847cdca-ae49-47a9-8dcc-a7e655fc2ad7","Type":"ContainerDied","Data":"45efd20996a533d8112b021a4f229b93b7b38c93ee2b266efc4eee85793abcd3"}
Nov 02 13:18:37.820476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.820442    5879 patch_prober.go:29] interesting pod/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body=
Nov 02 13:18:37.821036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.820501    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz" podUID="d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused"
Nov 02 13:18:37.851218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.851178    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"]
Nov 02 13:18:37.857447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:37.857408    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-25f7a6-openshift-default-5fcc86d7c-jlpqh"]
Nov 02 13:18:38.020296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:38.020215    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" path="/var/lib/kubelet/pods/8847cdca-ae49-47a9-8dcc-a7e655fc2ad7/volumes"
Nov 02 13:18:38.592734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:38.592702    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:38.649640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:38.649602    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"]
Nov 02 13:18:38.649863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:38.649811    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm" podUID="1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" containerName="discovery" containerID="cri-o://6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b" gracePeriod=30
Nov 02 13:18:38.819519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:38.819476    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:38.820011 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:38.819967    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:38.820809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:38.820784    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:18:38.988369 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:38.988337    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:18:39.071235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.071199    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-kube-api-access-2cftm\" (UniqueName: \"kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-kube-api-access-2cftm\") pod \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") "
Nov 02 13:18:39.071235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.071281    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-cacerts\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-cacerts\") pod \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") "
Nov 02 13:18:39.071235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.071360    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-token\" (UniqueName: \"kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-token\") pod \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") "
Nov 02 13:18:39.071235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.071441    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-local-certs\") pod \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") "
Nov 02 13:18:39.071235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.071509    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-kubeconfig\") pod \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") "
Nov 02 13:18:39.071235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.071634    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-ca-configmap\") pod \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") "
Nov 02 13:18:39.071235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.071716    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-dns-cert\") pod \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\" (UID: \"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b\") "
Nov 02 13:18:39.074705 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.074653    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-ca-configmap" pod "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" (UID: "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:18:39.076775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.076464    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-local-certs" pod "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" (UID: "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.076775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.076551    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-cacerts" pod "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" (UID: "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:18:39.078685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.078513    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-kubeconfig" pod "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" (UID: "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:18:39.078685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.078626    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-token" pod "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" (UID: "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:39.078836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.078697    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-dns-cert" pod "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" (UID: "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:18:39.079154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.079134    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-kube-api-access-2cftm" pod "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" (UID: "1afc3d86-7a0c-4f97-9298-204bd8bb0d2b"). InnerVolumeSpecName "kube-api-access-2cftm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:39.174339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.174302    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cftm\" (UniqueName: \"kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-kube-api-access-2cftm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.174339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.174332    5879 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-cacerts\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.174339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.174341    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.174339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.174349    5879 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-local-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.174630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.174358    5879 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-kubeconfig\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.174630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.174366    5879 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-ca-configmap\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.174630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.174376    5879 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b-istio-csr-dns-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.536872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.536843    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 13:18:39.579887 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.579857    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-kube-api-access-rlsgm\" (UniqueName: \"kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-kube-api-access-rlsgm\") pod \"5115dd72-ef9a-4a46-b84a-ee44b699082c\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") "
Nov 02 13:18:39.580105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.579959    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-token\" (UniqueName: \"kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-token\") pod \"5115dd72-ef9a-4a46-b84a-ee44b699082c\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") "
Nov 02 13:18:39.580179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.580119    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-socket\") pod \"5115dd72-ef9a-4a46-b84a-ee44b699082c\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") "
Nov 02 13:18:39.580238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.580226    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-credential-socket\") pod \"5115dd72-ef9a-4a46-b84a-ee44b699082c\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") "
Nov 02 13:18:39.580302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.580284    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-certs\") pod \"5115dd72-ef9a-4a46-b84a-ee44b699082c\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") "
Nov 02 13:18:39.581186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.580387    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-envoy\") pod \"5115dd72-ef9a-4a46-b84a-ee44b699082c\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") "
Nov 02 13:18:39.581186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.580454    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-data\") pod \"5115dd72-ef9a-4a46-b84a-ee44b699082c\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") "
Nov 02 13:18:39.581186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.580384    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-socket" pod "5115dd72-ef9a-4a46-b84a-ee44b699082c" (UID: "5115dd72-ef9a-4a46-b84a-ee44b699082c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.581186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.580482    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-credential-socket" pod "5115dd72-ef9a-4a46-b84a-ee44b699082c" (UID: "5115dd72-ef9a-4a46-b84a-ee44b699082c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.581186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.580529    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-podinfo\") pod \"5115dd72-ef9a-4a46-b84a-ee44b699082c\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") "
Nov 02 13:18:39.581186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.580607    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-certs" pod "5115dd72-ef9a-4a46-b84a-ee44b699082c" (UID: "5115dd72-ef9a-4a46-b84a-ee44b699082c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.581186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.580616    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5115dd72-ef9a-4a46-b84a-ee44b699082c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5115dd72-ef9a-4a46-b84a-ee44b699082c-istiod-ca-cert\") pod \"5115dd72-ef9a-4a46-b84a-ee44b699082c\" (UID: \"5115dd72-ef9a-4a46-b84a-ee44b699082c\") "
Nov 02 13:18:39.581186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.580715    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-data" pod "5115dd72-ef9a-4a46-b84a-ee44b699082c" (UID: "5115dd72-ef9a-4a46-b84a-ee44b699082c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.581186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.581152    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5115dd72-ef9a-4a46-b84a-ee44b699082c-istiod-ca-cert" pod "5115dd72-ef9a-4a46-b84a-ee44b699082c" (UID: "5115dd72-ef9a-4a46-b84a-ee44b699082c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:18:39.581794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.581584    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5115dd72-ef9a-4a46-b84a-ee44b699082c-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.581794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.581606    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.581794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.581621    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.581794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.581635    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.581794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.581649    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.582610 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.582590    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-envoy" pod "5115dd72-ef9a-4a46-b84a-ee44b699082c" (UID: "5115dd72-ef9a-4a46-b84a-ee44b699082c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:39.582675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.582650    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-token" pod "5115dd72-ef9a-4a46-b84a-ee44b699082c" (UID: "5115dd72-ef9a-4a46-b84a-ee44b699082c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:39.582932 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.582898    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-kube-api-access-rlsgm" pod "5115dd72-ef9a-4a46-b84a-ee44b699082c" (UID: "5115dd72-ef9a-4a46-b84a-ee44b699082c"). InnerVolumeSpecName "kube-api-access-rlsgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:39.583691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.583539    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-podinfo" pod "5115dd72-ef9a-4a46-b84a-ee44b699082c" (UID: "5115dd72-ef9a-4a46-b84a-ee44b699082c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:18:39.683604 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.683574    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.683604 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.683607    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.683758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.683617    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlsgm\" (UniqueName: \"kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-kube-api-access-rlsgm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.683758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.683626    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5115dd72-ef9a-4a46-b84a-ee44b699082c-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:39.727725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.727694    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:39.771786 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.771752    5879 generic.go:356] "Generic (PLEG): container finished" podID="5115dd72-ef9a-4a46-b84a-ee44b699082c" containerID="87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4" exitCode=0
Nov 02 13:18:39.772027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.771807    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"
Nov 02 13:18:39.772027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.771842    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" event={"ID":"5115dd72-ef9a-4a46-b84a-ee44b699082c","Type":"ContainerDied","Data":"87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4"}
Nov 02 13:18:39.772027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.771900    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr" event={"ID":"5115dd72-ef9a-4a46-b84a-ee44b699082c","Type":"ContainerDied","Data":"9c01eb1c7d1fbce8ab986fb95fa0657cb1e8a07d0ed1cafd0f54b1dac79458d2"}
Nov 02 13:18:39.772027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.771932    5879 scope.go:122] "RemoveContainer" containerID="87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4"
Nov 02 13:18:39.773608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.773570    5879 generic.go:356] "Generic (PLEG): container finished" podID="1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" containerID="6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b" exitCode=0
Nov 02 13:18:39.773740 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.773618    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"
Nov 02 13:18:39.773834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.773727    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm" event={"ID":"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b","Type":"ContainerDied","Data":"6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b"}
Nov 02 13:18:39.773897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.773860    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm" event={"ID":"1afc3d86-7a0c-4f97-9298-204bd8bb0d2b","Type":"ContainerDied","Data":"5301946cb76e3fbd216aa3a20b1d5ab72afe138b52a172f96134c5ffbfd350b0"}
Nov 02 13:18:39.784273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.784257    5879 scope.go:122] "RemoveContainer" containerID="87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4"
Nov 02 13:18:39.784579 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:18:39.784560    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4\": container with ID starting with 87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4 not found: ID does not exist" containerID="87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4"
Nov 02 13:18:39.784619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.784587    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4"} err="failed to get container status \"87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4\": rpc error: code = NotFound desc = could not find container \"87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4\": container with ID starting with 87ff63808fbe88531843a9419d2a1021a5732a818e8908ff18baae8fdc8c15e4 not found: ID does not exist"
Nov 02 13:18:39.784619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.784604    5879 scope.go:122] "RemoveContainer" containerID="6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b"
Nov 02 13:18:39.795696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.795677    5879 scope.go:122] "RemoveContainer" containerID="6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b"
Nov 02 13:18:39.795976 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:18:39.795957    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b\": container with ID starting with 6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b not found: ID does not exist" containerID="6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b"
Nov 02 13:18:39.796075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.795985    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b"} err="failed to get container status \"6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b\": rpc error: code = NotFound desc = could not find container \"6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b\": container with ID starting with 6898d7a327c0711bae1e9f629ae3ef7cd72fbca91552a8082e43685e0bd91f6b not found: ID does not exist"
Nov 02 13:18:39.893165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.893131    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"]
Nov 02 13:18:39.906468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.906439    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wr8nm"]
Nov 02 13:18:39.945863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.945826    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"]
Nov 02 13:18:39.950344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:39.950315    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-bc6xr"]
Nov 02 13:18:40.021986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:40.021942    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" path="/var/lib/kubelet/pods/1afc3d86-7a0c-4f97-9298-204bd8bb0d2b/volumes"
Nov 02 13:18:40.022845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:40.022822    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5115dd72-ef9a-4a46-b84a-ee44b699082c" path="/var/lib/kubelet/pods/5115dd72-ef9a-4a46-b84a-ee44b699082c/volumes"
Nov 02 13:18:40.350173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:40.350139    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:42.102392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:42.102332    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:43.008953 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.008927    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:18:43.143207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143166    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-credential-socket\") pod \"262814fe-6da4-48ae-85b8-c51aa9f2895a\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") "
Nov 02 13:18:43.143681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143234    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-data\") pod \"262814fe-6da4-48ae-85b8-c51aa9f2895a\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") "
Nov 02 13:18:43.143681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143266    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262814fe-6da4-48ae-85b8-c51aa9f2895a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262814fe-6da4-48ae-85b8-c51aa9f2895a-istiod-ca-cert\") pod \"262814fe-6da4-48ae-85b8-c51aa9f2895a\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") "
Nov 02 13:18:43.143681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143301    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-kube-api-access-ttnvb\" (UniqueName: \"kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-kube-api-access-ttnvb\") pod \"262814fe-6da4-48ae-85b8-c51aa9f2895a\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") "
Nov 02 13:18:43.143681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143347    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-envoy\") pod \"262814fe-6da4-48ae-85b8-c51aa9f2895a\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") "
Nov 02 13:18:43.143681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143387    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-token\" (UniqueName: \"kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-token\") pod \"262814fe-6da4-48ae-85b8-c51aa9f2895a\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") "
Nov 02 13:18:43.143681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143478    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-credential-socket" pod "262814fe-6da4-48ae-85b8-c51aa9f2895a" (UID: "262814fe-6da4-48ae-85b8-c51aa9f2895a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:43.143681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143491    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-podinfo\") pod \"262814fe-6da4-48ae-85b8-c51aa9f2895a\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") "
Nov 02 13:18:43.143681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143517    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-certs\") pod \"262814fe-6da4-48ae-85b8-c51aa9f2895a\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") "
Nov 02 13:18:43.143681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143541    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-socket\") pod \"262814fe-6da4-48ae-85b8-c51aa9f2895a\" (UID: \"262814fe-6da4-48ae-85b8-c51aa9f2895a\") "
Nov 02 13:18:43.143681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143542    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-data" pod "262814fe-6da4-48ae-85b8-c51aa9f2895a" (UID: "262814fe-6da4-48ae-85b8-c51aa9f2895a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:43.144514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143810    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-certs" pod "262814fe-6da4-48ae-85b8-c51aa9f2895a" (UID: "262814fe-6da4-48ae-85b8-c51aa9f2895a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:43.144514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.143894    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-socket" pod "262814fe-6da4-48ae-85b8-c51aa9f2895a" (UID: "262814fe-6da4-48ae-85b8-c51aa9f2895a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:43.144514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.144074    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262814fe-6da4-48ae-85b8-c51aa9f2895a-istiod-ca-cert" pod "262814fe-6da4-48ae-85b8-c51aa9f2895a" (UID: "262814fe-6da4-48ae-85b8-c51aa9f2895a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:18:43.144756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.144735    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:43.144833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.144762    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:43.144833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.144778    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262814fe-6da4-48ae-85b8-c51aa9f2895a-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:43.144833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.144793    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:43.144833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.144807    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:43.146162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.146132    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-envoy" pod "262814fe-6da4-48ae-85b8-c51aa9f2895a" (UID: "262814fe-6da4-48ae-85b8-c51aa9f2895a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:18:43.146162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.146149    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-token" pod "262814fe-6da4-48ae-85b8-c51aa9f2895a" (UID: "262814fe-6da4-48ae-85b8-c51aa9f2895a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:43.146332 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.146220    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-kube-api-access-ttnvb" pod "262814fe-6da4-48ae-85b8-c51aa9f2895a" (UID: "262814fe-6da4-48ae-85b8-c51aa9f2895a"). InnerVolumeSpecName "kube-api-access-ttnvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:18:43.146408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.146390    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-podinfo" pod "262814fe-6da4-48ae-85b8-c51aa9f2895a" (UID: "262814fe-6da4-48ae-85b8-c51aa9f2895a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:18:43.248258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.248170    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttnvb\" (UniqueName: \"kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-kube-api-access-ttnvb\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:43.248258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.248205    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:43.248258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.248219    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:43.248258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.248232    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262814fe-6da4-48ae-85b8-c51aa9f2895a-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:18:43.785593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.785551    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:43.812603 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.812559    5879 generic.go:356] "Generic (PLEG): container finished" podID="262814fe-6da4-48ae-85b8-c51aa9f2895a" containerID="82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941" exitCode=0
Nov 02 13:18:43.812821 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.812622    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"
Nov 02 13:18:43.812821 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.812616    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9" event={"ID":"262814fe-6da4-48ae-85b8-c51aa9f2895a","Type":"ContainerDied","Data":"82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941"}
Nov 02 13:18:43.812821 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.812748    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9" event={"ID":"262814fe-6da4-48ae-85b8-c51aa9f2895a","Type":"ContainerDied","Data":"44edf89609c365a31dce6a24a342b4f61751be8f06ad1ff89286bd32ac23bff9"}
Nov 02 13:18:43.812821 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.812778    5879 scope.go:122] "RemoveContainer" containerID="82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941"
Nov 02 13:18:43.826148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.826124    5879 scope.go:122] "RemoveContainer" containerID="82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941"
Nov 02 13:18:43.826461 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:18:43.826441    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941\": container with ID starting with 82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941 not found: ID does not exist" containerID="82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941"
Nov 02 13:18:43.826505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.826470    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941"} err="failed to get container status \"82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941\": rpc error: code = NotFound desc = could not find container \"82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941\": container with ID starting with 82a7b08622881597868da6cf267a8756186a77556c445f2dab384c40aca2b941 not found: ID does not exist"
Nov 02 13:18:43.873647 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.873614    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"]
Nov 02 13:18:43.886788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:43.886759    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-95ee26-openshift-default-66679d8f57-67sz9"]
Nov 02 13:18:44.020711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:44.020672    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262814fe-6da4-48ae-85b8-c51aa9f2895a" path="/var/lib/kubelet/pods/262814fe-6da4-48ae-85b8-c51aa9f2895a/volumes"
Nov 02 13:18:44.825841 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:18:44.825802    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:18:45.470592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:45.470554    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:46.990471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:46.990440    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:48.738437 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:48.738400    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:50.305303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:50.305256    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:51.845663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:51.845627    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:53.288751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.288701    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"]
Nov 02 13:18:53.289631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.289608    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5115dd72-ef9a-4a46-b84a-ee44b699082c" containerName="istio-proxy"
Nov 02 13:18:53.289631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.289628    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5115dd72-ef9a-4a46-b84a-ee44b699082c" containerName="istio-proxy"
Nov 02 13:18:53.289808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.289651    5879 cpu_manager.go:414] "containerMap: removing container" podUID="8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" containerName="istio-proxy"
Nov 02 13:18:53.289808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.289657    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" containerName="istio-proxy"
Nov 02 13:18:53.289808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.289671    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" containerName="discovery"
Nov 02 13:18:53.289808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.289676    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" containerName="discovery"
Nov 02 13:18:53.289808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.289685    5879 cpu_manager.go:414] "containerMap: removing container" podUID="262814fe-6da4-48ae-85b8-c51aa9f2895a" containerName="istio-proxy"
Nov 02 13:18:53.289808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.289690    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262814fe-6da4-48ae-85b8-c51aa9f2895a" containerName="istio-proxy"
Nov 02 13:18:53.290074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.289833    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5115dd72-ef9a-4a46-b84a-ee44b699082c" containerName="istio-proxy"
Nov 02 13:18:53.290074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.289842    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="8847cdca-ae49-47a9-8dcc-a7e655fc2ad7" containerName="istio-proxy"
Nov 02 13:18:53.290074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.289849    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1afc3d86-7a0c-4f97-9298-204bd8bb0d2b" containerName="discovery"
Nov 02 13:18:53.290074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.289856    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="262814fe-6da4-48ae-85b8-c51aa9f2895a" containerName="istio-proxy"
Nov 02 13:18:53.296328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.296307    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.313248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.313211    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9139ec-openshift-default-dockercfg-4ltv6\""
Nov 02 13:18:53.332636 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.332601    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"]
Nov 02 13:18:53.411059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.410969    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-socket\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.411250 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.411196    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-data\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.411294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.411262    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7768beb-47c2-4a5d-a281-75e09a47965b-istiod-ca-cert\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.411380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.411357    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-certs\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.411447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.411430    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-podinfo\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.411513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.411498    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-envoy\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.411690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.411668    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-credential-socket\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.411745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.411729    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92j94\" (UniqueName: \"kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-kube-api-access-92j94\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.411942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.411924    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-token\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.514084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.514040    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-credential-socket\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.514308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.514106    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92j94\" (UniqueName: \"kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-kube-api-access-92j94\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.514308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.514214    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-token\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.514308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.514242    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-socket\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.514492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.514316    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-data\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.514492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.514347    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7768beb-47c2-4a5d-a281-75e09a47965b-istiod-ca-cert\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.514492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.514378    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-certs\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.514492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.514408    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-podinfo\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.514492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.514443    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-envoy\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.515223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.515193    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-socket\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.515427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.515252    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-data\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.515427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.515305    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-certs\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.515576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.515348    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-credential-socket\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.516057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.516029    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7768beb-47c2-4a5d-a281-75e09a47965b-istiod-ca-cert\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.517434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.517413    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-envoy\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.517859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.517841    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-podinfo\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.540628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.540563    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-token\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.540775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.540754    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92j94\" (UniqueName: \"kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-kube-api-access-92j94\") pod \"e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") " pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.611856 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.611824    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:53.737241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.737202    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:53.943888 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:18:53.943855    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7768beb_47c2_4a5d_a281_75e09a47965b.slice/crio-bf894c355dfb6bc3937c1038a7aae2de68094cc1aeb067dbd5cee690ca31b9d0 WatchSource:0}: Error finding container bf894c355dfb6bc3937c1038a7aae2de68094cc1aeb067dbd5cee690ca31b9d0: Status 404 returned error can't find the container with id bf894c355dfb6bc3937c1038a7aae2de68094cc1aeb067dbd5cee690ca31b9d0
Nov 02 13:18:53.945840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.945811    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"]
Nov 02 13:18:53.947036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.946952    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:18:53.947180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.947050    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:18:53.947180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:53.947097    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:18:54.924030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:54.923977    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl" event={"ID":"f7768beb-47c2-4a5d-a281-75e09a47965b","Type":"ContainerStarted","Data":"8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b"}
Nov 02 13:18:54.924456 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:54.924060    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl" event={"ID":"f7768beb-47c2-4a5d-a281-75e09a47965b","Type":"ContainerStarted","Data":"bf894c355dfb6bc3937c1038a7aae2de68094cc1aeb067dbd5cee690ca31b9d0"}
Nov 02 13:18:55.008209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:55.008137    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl" podStartSLOduration=2.008114904 podStartE2EDuration="2.008114904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:18:55.007424472 +0000 UTC m=+3081.114127337" watchObservedRunningTime="2026-11-02 13:18:55.008114904 +0000 UTC m=+3081.114817768"
Nov 02 13:18:55.468580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:55.468547    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:55.612218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:55.612177    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:18:55.613495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:55.613469    5879 patch_prober.go:29] interesting pod/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" start-of-body=
Nov 02 13:18:55.613646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:55.613545    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl" podUID="f7768beb-47c2-4a5d-a281-75e09a47965b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused"
Nov 02 13:18:56.612443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:56.612405    5879 patch_prober.go:29] interesting pod/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" start-of-body=
Nov 02 13:18:56.612850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:56.612467    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl" podUID="f7768beb-47c2-4a5d-a281-75e09a47965b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused"
Nov 02 13:18:57.122367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:57.122334    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:57.612915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:57.612806    5879 patch_prober.go:29] interesting pod/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" start-of-body=
Nov 02 13:18:57.612915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:57.612873    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl" podUID="f7768beb-47c2-4a5d-a281-75e09a47965b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused"
Nov 02 13:18:57.942861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:57.942825    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:58.369470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:58.369425    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:58.612738 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:58.612706    5879 patch_prober.go:29] interesting pod/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" start-of-body=
Nov 02 13:18:58.612929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:58.612764    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl" podUID="f7768beb-47c2-4a5d-a281-75e09a47965b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused"
Nov 02 13:18:58.833970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:58.833874    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:58.983507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:58.983473    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:59.109413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:59.109319    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:59.249271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:59.249196    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:18:59.612541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:59.612499    5879 patch_prober.go:29] interesting pod/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused" start-of-body=
Nov 02 13:18:59.612781 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:18:59.612568    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl" podUID="f7768beb-47c2-4a5d-a281-75e09a47965b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.77:15021/healthz/ready\": dial tcp 10.134.1.77:15021: connect: connection refused"
Nov 02 13:19:00.531163 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.531128    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"]
Nov 02 13:19:00.537977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.537948    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:00.573354 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.573319    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"]
Nov 02 13:19:00.575425 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.575336    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:19:00.575732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.575711    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:19:00.686405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.684832    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:19:00.686405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.686400    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:19:00.688321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.688280    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:19:00.693537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.693503    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dea6e7a-zrlsg\" (UID: \"d4068480-4ce9-4956-ab43-22166dfcccc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:00.693748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.693558    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dea6e7a-zrlsg\" (UID: \"d4068480-4ce9-4956-ab43-22166dfcccc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:00.693838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.693820    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x97p7\" (UniqueName: \"kubernetes.io/projected/d4068480-4ce9-4956-ab43-22166dfcccc1-kube-api-access-x97p7\") pod \"maas-api-revoke-sub-4dea6e7a-zrlsg\" (UID: \"d4068480-4ce9-4956-ab43-22166dfcccc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:00.794945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.794853    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97p7\" (UniqueName: \"kubernetes.io/projected/d4068480-4ce9-4956-ab43-22166dfcccc1-kube-api-access-x97p7\") pod \"maas-api-revoke-sub-4dea6e7a-zrlsg\" (UID: \"d4068480-4ce9-4956-ab43-22166dfcccc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:00.795181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.795071    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dea6e7a-zrlsg\" (UID: \"d4068480-4ce9-4956-ab43-22166dfcccc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:00.795181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.795112    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dea6e7a-zrlsg\" (UID: \"d4068480-4ce9-4956-ab43-22166dfcccc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:00.796140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.796114    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4dea6e7a-zrlsg\" (UID: \"d4068480-4ce9-4956-ab43-22166dfcccc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:00.799352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.799318    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4dea6e7a-zrlsg\" (UID: \"d4068480-4ce9-4956-ab43-22166dfcccc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:00.821125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.821096    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97p7\" (UniqueName: \"kubernetes.io/projected/d4068480-4ce9-4956-ab43-22166dfcccc1-kube-api-access-x97p7\") pod \"maas-api-revoke-sub-4dea6e7a-zrlsg\" (UID: \"d4068480-4ce9-4956-ab43-22166dfcccc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:00.875297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:00.875269    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:01.089667 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:01.089575    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:01.191888 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:19:01.191854    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4068480_4ce9_4956_ab43_22166dfcccc1.slice/crio-90f71a8f5a552ddfeab46072c6dc146bf3d88a127444afc64f1f058e371c8e50 WatchSource:0}: Error finding container 90f71a8f5a552ddfeab46072c6dc146bf3d88a127444afc64f1f058e371c8e50: Status 404 returned error can't find the container with id 90f71a8f5a552ddfeab46072c6dc146bf3d88a127444afc64f1f058e371c8e50
Nov 02 13:19:01.193819 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:01.193743    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"]
Nov 02 13:19:02.094219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:02.094180    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg" event={"ID":"d4068480-4ce9-4956-ab43-22166dfcccc1","Type":"ContainerStarted","Data":"7dab98508a0f87cd0721b1aa12a4add52cad9dbd80403cefbddddebf5a5cf072"}
Nov 02 13:19:02.094219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:02.094237    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg" event={"ID":"d4068480-4ce9-4956-ab43-22166dfcccc1","Type":"ContainerStarted","Data":"90f71a8f5a552ddfeab46072c6dc146bf3d88a127444afc64f1f058e371c8e50"}
Nov 02 13:19:02.147288 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:02.147228    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg" podStartSLOduration=2.147210894 podStartE2EDuration="2.147210894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:19:02.143944084 +0000 UTC m=+3088.250646937" watchObservedRunningTime="2026-11-02 13:19:02.147210894 +0000 UTC m=+3088.253913745"
Nov 02 13:19:02.952597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:02.952562    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:03.164225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:03.164179    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:03.174741 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:03.174707    5879 generic.go:356] "Generic (PLEG): container finished" podID="d4068480-4ce9-4956-ab43-22166dfcccc1" containerID="7dab98508a0f87cd0721b1aa12a4add52cad9dbd80403cefbddddebf5a5cf072" exitCode=0
Nov 02 13:19:03.175182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:03.175158    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg" event={"ID":"d4068480-4ce9-4956-ab43-22166dfcccc1","Type":"ContainerDied","Data":"7dab98508a0f87cd0721b1aa12a4add52cad9dbd80403cefbddddebf5a5cf072"}
Nov 02 13:19:04.359379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.359354    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:04.385374 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.385343    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4068480-4ce9-4956-ab43-22166dfcccc1-kube-api-access-x97p7\" (UniqueName: \"kubernetes.io/projected/d4068480-4ce9-4956-ab43-22166dfcccc1-kube-api-access-x97p7\") pod \"d4068480-4ce9-4956-ab43-22166dfcccc1\" (UID: \"d4068480-4ce9-4956-ab43-22166dfcccc1\") "
Nov 02 13:19:04.385579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.385429    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-openshift-ca\") pod \"d4068480-4ce9-4956-ab43-22166dfcccc1\" (UID: \"d4068480-4ce9-4956-ab43-22166dfcccc1\") "
Nov 02 13:19:04.385817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.385793    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-xks-ca\") pod \"d4068480-4ce9-4956-ab43-22166dfcccc1\" (UID: \"d4068480-4ce9-4956-ab43-22166dfcccc1\") "
Nov 02 13:19:04.385894 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.385851    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-openshift-ca" pod "d4068480-4ce9-4956-ab43-22166dfcccc1" (UID: "d4068480-4ce9-4956-ab43-22166dfcccc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:19:04.386777 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.386753    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:04.387928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.387903    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4068480-4ce9-4956-ab43-22166dfcccc1-kube-api-access-x97p7" pod "d4068480-4ce9-4956-ab43-22166dfcccc1" (UID: "d4068480-4ce9-4956-ab43-22166dfcccc1"). InnerVolumeSpecName "kube-api-access-x97p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:19:04.388057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.388009    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-xks-ca" pod "d4068480-4ce9-4956-ab43-22166dfcccc1" (UID: "d4068480-4ce9-4956-ab43-22166dfcccc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:19:04.488393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.488300    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4068480-4ce9-4956-ab43-22166dfcccc1-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:04.488393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.488334    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x97p7\" (UniqueName: \"kubernetes.io/projected/d4068480-4ce9-4956-ab43-22166dfcccc1-kube-api-access-x97p7\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:04.696694 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.696667    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:04.835953 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.835864    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"]
Nov 02 13:19:04.836895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.836858    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d4068480-4ce9-4956-ab43-22166dfcccc1" containerName="revoke-keys"
Nov 02 13:19:04.836895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.836881    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4068480-4ce9-4956-ab43-22166dfcccc1" containerName="revoke-keys"
Nov 02 13:19:04.837120 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.837105    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4068480-4ce9-4956-ab43-22166dfcccc1" containerName="revoke-keys"
Nov 02 13:19:04.841190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.841166    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"
Nov 02 13:19:04.858480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.858444    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-357b30\""
Nov 02 13:19:04.861692 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.861667    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-357b30-dockercfg-qsfk9\""
Nov 02 13:19:04.862434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.862402    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:04.863483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.863463    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"]
Nov 02 13:19:04.893964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.893929    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb2c475-7933-4442-8df2-91761ce122aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8\" (UID: \"1eb2c475-7933-4442-8df2-91761ce122aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"
Nov 02 13:19:04.894276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.894251    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8djht\" (UniqueName: \"kubernetes.io/projected/1eb2c475-7933-4442-8df2-91761ce122aa-kube-api-access-8djht\") pod \"payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8\" (UID: \"1eb2c475-7933-4442-8df2-91761ce122aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"
Nov 02 13:19:05.018117 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:04.999123    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8djht\" (UniqueName: \"kubernetes.io/projected/1eb2c475-7933-4442-8df2-91761ce122aa-kube-api-access-8djht\") pod \"payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8\" (UID: \"1eb2c475-7933-4442-8df2-91761ce122aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"
Nov 02 13:19:05.018117 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.006344    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb2c475-7933-4442-8df2-91761ce122aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8\" (UID: \"1eb2c475-7933-4442-8df2-91761ce122aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"
Nov 02 13:19:05.018117 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.008646    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb2c475-7933-4442-8df2-91761ce122aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8\" (UID: \"1eb2c475-7933-4442-8df2-91761ce122aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"
Nov 02 13:19:05.018117 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.013190    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"]
Nov 02 13:19:05.090452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.081132    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:05.104758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.104069    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"]
Nov 02 13:19:05.104758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.104228    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"
Nov 02 13:19:05.105883 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.105856    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8djht\" (UniqueName: \"kubernetes.io/projected/1eb2c475-7933-4442-8df2-91761ce122aa-kube-api-access-8djht\") pod \"payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8\" (UID: \"1eb2c475-7933-4442-8df2-91761ce122aa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"
Nov 02 13:19:05.156742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.156709    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"
Nov 02 13:19:05.194983 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.194933    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"
Nov 02 13:19:05.194983 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.194941    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg" event={"ID":"d4068480-4ce9-4956-ab43-22166dfcccc1","Type":"ContainerDied","Data":"90f71a8f5a552ddfeab46072c6dc146bf3d88a127444afc64f1f058e371c8e50"}
Nov 02 13:19:05.195240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.195106    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90f71a8f5a552ddfeab46072c6dc146bf3d88a127444afc64f1f058e371c8e50"
Nov 02 13:19:05.210755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.210719    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf747cbc-bea2-4edd-97d2-562ad019abb0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk\" (UID: \"cf747cbc-bea2-4edd-97d2-562ad019abb0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"
Nov 02 13:19:05.210958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.210774    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmrhr\" (UniqueName: \"kubernetes.io/projected/cf747cbc-bea2-4edd-97d2-562ad019abb0-kube-api-access-rmrhr\") pod \"payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk\" (UID: \"cf747cbc-bea2-4edd-97d2-562ad019abb0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"
Nov 02 13:19:05.243458 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.243425    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:05.312530 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.312496    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf747cbc-bea2-4edd-97d2-562ad019abb0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk\" (UID: \"cf747cbc-bea2-4edd-97d2-562ad019abb0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"
Nov 02 13:19:05.312742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.312549    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmrhr\" (UniqueName: \"kubernetes.io/projected/cf747cbc-bea2-4edd-97d2-562ad019abb0-kube-api-access-rmrhr\") pod \"payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk\" (UID: \"cf747cbc-bea2-4edd-97d2-562ad019abb0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"
Nov 02 13:19:05.313237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.313218    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf747cbc-bea2-4edd-97d2-562ad019abb0-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk\" (UID: \"cf747cbc-bea2-4edd-97d2-562ad019abb0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"
Nov 02 13:19:05.346255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.346182    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmrhr\" (UniqueName: \"kubernetes.io/projected/cf747cbc-bea2-4edd-97d2-562ad019abb0-kube-api-access-rmrhr\") pod \"payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk\" (UID: \"cf747cbc-bea2-4edd-97d2-562ad019abb0\") " pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"
Nov 02 13:19:05.438585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.438555    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"
Nov 02 13:19:05.486225 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:19:05.485875    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb2c475_7933_4442_8df2_91761ce122aa.slice/crio-192c9509b2721e1129714b7293af5703304c5c1771f457cbc81c43cd81a6f09c WatchSource:0}: Error finding container 192c9509b2721e1129714b7293af5703304c5c1771f457cbc81c43cd81a6f09c: Status 404 returned error can't find the container with id 192c9509b2721e1129714b7293af5703304c5c1771f457cbc81c43cd81a6f09c
Nov 02 13:19:05.488851 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.488823    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"]
Nov 02 13:19:05.760567 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:19:05.760532    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf747cbc_bea2_4edd_97d2_562ad019abb0.slice/crio-5e859d763eea5ce57c3031c580536c738f8db86203f5596487f7752592d246af WatchSource:0}: Error finding container 5e859d763eea5ce57c3031c580536c738f8db86203f5596487f7752592d246af: Status 404 returned error can't find the container with id 5e859d763eea5ce57c3031c580536c738f8db86203f5596487f7752592d246af
Nov 02 13:19:05.762868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:05.762846    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"]
Nov 02 13:19:06.206344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:06.206296    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk" event={"ID":"cf747cbc-bea2-4edd-97d2-562ad019abb0","Type":"ContainerStarted","Data":"d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075"}
Nov 02 13:19:06.206537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:06.206401    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"
Nov 02 13:19:06.206537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:06.206448    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk" event={"ID":"cf747cbc-bea2-4edd-97d2-562ad019abb0","Type":"ContainerStarted","Data":"5e859d763eea5ce57c3031c580536c738f8db86203f5596487f7752592d246af"}
Nov 02 13:19:06.208550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:06.208507    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8" event={"ID":"1eb2c475-7933-4442-8df2-91761ce122aa","Type":"ContainerStarted","Data":"a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395"}
Nov 02 13:19:06.208674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:06.208579    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"
Nov 02 13:19:06.208674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:06.208599    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8" event={"ID":"1eb2c475-7933-4442-8df2-91761ce122aa","Type":"ContainerStarted","Data":"192c9509b2721e1129714b7293af5703304c5c1771f457cbc81c43cd81a6f09c"}
Nov 02 13:19:06.256127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:06.256074    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk" podStartSLOduration=2.256057064 podStartE2EDuration="2.256057064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:19:06.25016006 +0000 UTC m=+3092.356862912" watchObservedRunningTime="2026-11-02 13:19:06.256057064 +0000 UTC m=+3092.362759962"
Nov 02 13:19:06.298751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:06.298690    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8" podStartSLOduration=2.298672715 podStartE2EDuration="2.298672715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:19:06.295341438 +0000 UTC m=+3092.402044289" watchObservedRunningTime="2026-11-02 13:19:06.298672715 +0000 UTC m=+3092.405375566"
Nov 02 13:19:06.667793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:06.667765    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:08.265236 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:08.265201    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:09.900725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:09.900681    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:11.407634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:11.405745    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"]
Nov 02 13:19:11.474710 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:11.474663    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:11.616648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:11.616440    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:11.927392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:11.927351    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"]
Nov 02 13:19:11.932816 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:11.932786    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:11.953109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:11.953080    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"]
Nov 02 13:19:11.962414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:11.962383    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:19:11.962414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:11.962406    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:19:12.023177 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.023142    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49b6c\" (UniqueName: \"kubernetes.io/projected/25175e92-cbb7-46b6-b261-190631c4f98d-kube-api-access-49b6c\") pod \"maas-api-revoke-sub-e73f12ba-25jlq\" (UID: \"25175e92-cbb7-46b6-b261-190631c4f98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:12.023406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.023383    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e73f12ba-25jlq\" (UID: \"25175e92-cbb7-46b6-b261-190631c4f98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:12.023513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.023502    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e73f12ba-25jlq\" (UID: \"25175e92-cbb7-46b6-b261-190631c4f98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:12.125755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.125712    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e73f12ba-25jlq\" (UID: \"25175e92-cbb7-46b6-b261-190631c4f98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:12.125986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.125893    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e73f12ba-25jlq\" (UID: \"25175e92-cbb7-46b6-b261-190631c4f98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:12.125986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.125977    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49b6c\" (UniqueName: \"kubernetes.io/projected/25175e92-cbb7-46b6-b261-190631c4f98d-kube-api-access-49b6c\") pod \"maas-api-revoke-sub-e73f12ba-25jlq\" (UID: \"25175e92-cbb7-46b6-b261-190631c4f98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:12.126563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.126538    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e73f12ba-25jlq\" (UID: \"25175e92-cbb7-46b6-b261-190631c4f98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:12.128399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.128377    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e73f12ba-25jlq\" (UID: \"25175e92-cbb7-46b6-b261-190631c4f98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:12.151609 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.151571    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49b6c\" (UniqueName: \"kubernetes.io/projected/25175e92-cbb7-46b6-b261-190631c4f98d-kube-api-access-49b6c\") pod \"maas-api-revoke-sub-e73f12ba-25jlq\" (UID: \"25175e92-cbb7-46b6-b261-190631c4f98d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:12.249167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.249074    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:12.568218 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:19:12.568187    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25175e92_cbb7_46b6_b261_190631c4f98d.slice/crio-c070ac59a2512232f3c91b706273ace55ea52b639df1dc89dff3b5ada65fd19f WatchSource:0}: Error finding container c070ac59a2512232f3c91b706273ace55ea52b639df1dc89dff3b5ada65fd19f: Status 404 returned error can't find the container with id c070ac59a2512232f3c91b706273ace55ea52b639df1dc89dff3b5ada65fd19f
Nov 02 13:19:12.569700 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.569670    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"]
Nov 02 13:19:12.610417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.610388    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"]
Nov 02 13:19:12.610608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:12.610588    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy" containerID="cri-o://e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7" gracePeriod=30
Nov 02 13:19:13.266734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:13.266645    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:13.271429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:13.271394    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq" event={"ID":"25175e92-cbb7-46b6-b261-190631c4f98d","Type":"ContainerStarted","Data":"d7e3ebe43ca793219fc6f9c0acd816226fc3d46a40670c9aefa996180a2ec5b9"}
Nov 02 13:19:13.271685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:13.271661    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq" event={"ID":"25175e92-cbb7-46b6-b261-190631c4f98d","Type":"ContainerStarted","Data":"c070ac59a2512232f3c91b706273ace55ea52b639df1dc89dff3b5ada65fd19f"}
Nov 02 13:19:13.316865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:13.316115    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq" podStartSLOduration=2.316097677 podStartE2EDuration="2.316097677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:19:13.315779721 +0000 UTC m=+3099.422482574" watchObservedRunningTime="2026-11-02 13:19:13.316097677 +0000 UTC m=+3099.422800528"
Nov 02 13:19:14.281970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:14.281930    5879 generic.go:356] "Generic (PLEG): container finished" podID="25175e92-cbb7-46b6-b261-190631c4f98d" containerID="d7e3ebe43ca793219fc6f9c0acd816226fc3d46a40670c9aefa996180a2ec5b9" exitCode=0
Nov 02 13:19:14.282398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:14.281985    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq" event={"ID":"25175e92-cbb7-46b6-b261-190631c4f98d","Type":"ContainerDied","Data":"d7e3ebe43ca793219fc6f9c0acd816226fc3d46a40670c9aefa996180a2ec5b9"}
Nov 02 13:19:14.823845 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:19:14.823803    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:19:14.902313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:14.902280    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:15.472635 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:15.472609    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:15.583105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:15.582887    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-xks-ca\") pod \"25175e92-cbb7-46b6-b261-190631c4f98d\" (UID: \"25175e92-cbb7-46b6-b261-190631c4f98d\") "
Nov 02 13:19:15.583105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:15.583029    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25175e92-cbb7-46b6-b261-190631c4f98d-kube-api-access-49b6c\" (UniqueName: \"kubernetes.io/projected/25175e92-cbb7-46b6-b261-190631c4f98d-kube-api-access-49b6c\") pod \"25175e92-cbb7-46b6-b261-190631c4f98d\" (UID: \"25175e92-cbb7-46b6-b261-190631c4f98d\") "
Nov 02 13:19:15.583356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:15.583274    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-openshift-ca\") pod \"25175e92-cbb7-46b6-b261-190631c4f98d\" (UID: \"25175e92-cbb7-46b6-b261-190631c4f98d\") "
Nov 02 13:19:15.583843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:15.583811    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-openshift-ca" pod "25175e92-cbb7-46b6-b261-190631c4f98d" (UID: "25175e92-cbb7-46b6-b261-190631c4f98d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:19:15.584136 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:15.584117    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:15.585838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:15.585806    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-xks-ca" pod "25175e92-cbb7-46b6-b261-190631c4f98d" (UID: "25175e92-cbb7-46b6-b261-190631c4f98d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:19:15.585971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:15.585871    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25175e92-cbb7-46b6-b261-190631c4f98d-kube-api-access-49b6c" pod "25175e92-cbb7-46b6-b261-190631c4f98d" (UID: "25175e92-cbb7-46b6-b261-190631c4f98d"). InnerVolumeSpecName "kube-api-access-49b6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:19:15.685790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:15.685751    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25175e92-cbb7-46b6-b261-190631c4f98d-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:15.685790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:15.685783    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49b6c\" (UniqueName: \"kubernetes.io/projected/25175e92-cbb7-46b6-b261-190631c4f98d-kube-api-access-49b6c\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:16.301320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:16.301215    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"
Nov 02 13:19:16.301320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:16.301223    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq" event={"ID":"25175e92-cbb7-46b6-b261-190631c4f98d","Type":"ContainerDied","Data":"c070ac59a2512232f3c91b706273ace55ea52b639df1dc89dff3b5ada65fd19f"}
Nov 02 13:19:16.301521 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:16.301349    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c070ac59a2512232f3c91b706273ace55ea52b639df1dc89dff3b5ada65fd19f"
Nov 02 13:19:16.434120 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:16.434084    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:17.220317 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.220288    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"
Nov 02 13:19:17.220794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.220366    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"
Nov 02 13:19:17.910126 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.910053    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:19:17.940561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.940527    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-kube-api-access-sv68r\" (UniqueName: \"kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-kube-api-access-sv68r\") pod \"31e95cd7-e5a6-446f-90c5-db47c0efe379\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") "
Nov 02 13:19:17.940769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.940590    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-envoy\") pod \"31e95cd7-e5a6-446f-90c5-db47c0efe379\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") "
Nov 02 13:19:17.940769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.940715    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31e95cd7-e5a6-446f-90c5-db47c0efe379-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e95cd7-e5a6-446f-90c5-db47c0efe379-istiod-ca-cert\") pod \"31e95cd7-e5a6-446f-90c5-db47c0efe379\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") "
Nov 02 13:19:17.940900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.940833    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-data\") pod \"31e95cd7-e5a6-446f-90c5-db47c0efe379\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") "
Nov 02 13:19:17.940900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.940886    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-credential-socket\") pod \"31e95cd7-e5a6-446f-90c5-db47c0efe379\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") "
Nov 02 13:19:17.941046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.940925    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-token\" (UniqueName: \"kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-token\") pod \"31e95cd7-e5a6-446f-90c5-db47c0efe379\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") "
Nov 02 13:19:17.941046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.941030    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-certs\") pod \"31e95cd7-e5a6-446f-90c5-db47c0efe379\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") "
Nov 02 13:19:17.941167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.941099    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-socket\") pod \"31e95cd7-e5a6-446f-90c5-db47c0efe379\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") "
Nov 02 13:19:17.941167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.941135    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-podinfo\") pod \"31e95cd7-e5a6-446f-90c5-db47c0efe379\" (UID: \"31e95cd7-e5a6-446f-90c5-db47c0efe379\") "
Nov 02 13:19:17.941276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.941196    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31e95cd7-e5a6-446f-90c5-db47c0efe379-istiod-ca-cert" pod "31e95cd7-e5a6-446f-90c5-db47c0efe379" (UID: "31e95cd7-e5a6-446f-90c5-db47c0efe379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:19:17.941661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.941568    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-certs" pod "31e95cd7-e5a6-446f-90c5-db47c0efe379" (UID: "31e95cd7-e5a6-446f-90c5-db47c0efe379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:17.942236 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.942210    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31e95cd7-e5a6-446f-90c5-db47c0efe379-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:17.942350 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.942242    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:17.942640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.942616    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-data" pod "31e95cd7-e5a6-446f-90c5-db47c0efe379" (UID: "31e95cd7-e5a6-446f-90c5-db47c0efe379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:17.942791 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.942769    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-credential-socket" pod "31e95cd7-e5a6-446f-90c5-db47c0efe379" (UID: "31e95cd7-e5a6-446f-90c5-db47c0efe379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:17.942857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.942794    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-socket" pod "31e95cd7-e5a6-446f-90c5-db47c0efe379" (UID: "31e95cd7-e5a6-446f-90c5-db47c0efe379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:17.943617 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.943588    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-envoy" pod "31e95cd7-e5a6-446f-90c5-db47c0efe379" (UID: "31e95cd7-e5a6-446f-90c5-db47c0efe379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:17.943774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.943746    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-kube-api-access-sv68r" pod "31e95cd7-e5a6-446f-90c5-db47c0efe379" (UID: "31e95cd7-e5a6-446f-90c5-db47c0efe379"). InnerVolumeSpecName "kube-api-access-sv68r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:19:17.944367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.944345    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-token" pod "31e95cd7-e5a6-446f-90c5-db47c0efe379" (UID: "31e95cd7-e5a6-446f-90c5-db47c0efe379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:19:17.944612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:17.944596    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-podinfo" pod "31e95cd7-e5a6-446f-90c5-db47c0efe379" (UID: "31e95cd7-e5a6-446f-90c5-db47c0efe379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:19:18.044405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.044378    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:18.044405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.044402    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:18.044405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.044412    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:18.044659 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.044447    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:18.044659 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.044469    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv68r\" (UniqueName: \"kubernetes.io/projected/31e95cd7-e5a6-446f-90c5-db47c0efe379-kube-api-access-sv68r\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:18.044659 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.044483    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:18.044659 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.044494    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31e95cd7-e5a6-446f-90c5-db47c0efe379-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:18.325929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.325825    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:18.474394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.474355    5879 generic.go:356] "Generic (PLEG): container finished" podID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerID="e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7" exitCode=0
Nov 02 13:19:18.474608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.474424    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"
Nov 02 13:19:18.474608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.474433    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" event={"ID":"31e95cd7-e5a6-446f-90c5-db47c0efe379","Type":"ContainerDied","Data":"e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7"}
Nov 02 13:19:18.474608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.474487    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7" event={"ID":"31e95cd7-e5a6-446f-90c5-db47c0efe379","Type":"ContainerDied","Data":"b955bdbc30598258195b4809c3d36326d2c2d46c43cdff16036c87f37bb337b2"}
Nov 02 13:19:18.474608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.474508    5879 scope.go:122] "RemoveContainer" containerID="e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7"
Nov 02 13:19:18.486633 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.486609    5879 scope.go:122] "RemoveContainer" containerID="e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7"
Nov 02 13:19:18.486927 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:19:18.486898    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7\": container with ID starting with e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7 not found: ID does not exist" containerID="e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7"
Nov 02 13:19:18.486977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.486935    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7"} err="failed to get container status \"e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7\": rpc error: code = NotFound desc = could not find container \"e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7\": container with ID starting with e38438b242e779b5f2d50b6130606c3a59208ef5fed968e1a98197f7445baee7 not found: ID does not exist"
Nov 02 13:19:18.534350 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.534317    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"]
Nov 02 13:19:18.548461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:18.548428    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-sg8t7"]
Nov 02 13:19:20.021936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:20.021898    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" path="/var/lib/kubelet/pods/31e95cd7-e5a6-446f-90c5-db47c0efe379/volumes"
Nov 02 13:19:20.253431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:20.253354    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:21.806774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:21.806739    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:23.758556 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:23.758478    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:25.437696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:25.437646    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:26.439065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.439022    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg" podUID="0f015abe-55db-486b-a074-43553ea9e791" containerName="main" containerID="cri-o://18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4" gracePeriod=45
Nov 02 13:19:26.746768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.746737    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:19:26.908182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.908153    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-home\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-home\") pod \"0f015abe-55db-486b-a074-43553ea9e791\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") "
Nov 02 13:19:26.908358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.908326    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f015abe-55db-486b-a074-43553ea9e791-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f015abe-55db-486b-a074-43553ea9e791-tls-certs\") pod \"0f015abe-55db-486b-a074-43553ea9e791\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") "
Nov 02 13:19:26.908358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.908349    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-tmp-dir\") pod \"0f015abe-55db-486b-a074-43553ea9e791\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") "
Nov 02 13:19:26.908474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.908394    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-model-cache\") pod \"0f015abe-55db-486b-a074-43553ea9e791\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") "
Nov 02 13:19:26.908474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.908431    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-home" pod "0f015abe-55db-486b-a074-43553ea9e791" (UID: "0f015abe-55db-486b-a074-43553ea9e791"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:26.908592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.908480    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-dshm\") pod \"0f015abe-55db-486b-a074-43553ea9e791\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") "
Nov 02 13:19:26.908592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.908514    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f015abe-55db-486b-a074-43553ea9e791-kube-api-access-hnvzx\" (UniqueName: \"kubernetes.io/projected/0f015abe-55db-486b-a074-43553ea9e791-kube-api-access-hnvzx\") pod \"0f015abe-55db-486b-a074-43553ea9e791\" (UID: \"0f015abe-55db-486b-a074-43553ea9e791\") "
Nov 02 13:19:26.908707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.908666    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-model-cache" pod "0f015abe-55db-486b-a074-43553ea9e791" (UID: "0f015abe-55db-486b-a074-43553ea9e791"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:26.908707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.908684    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-tmp-dir" pod "0f015abe-55db-486b-a074-43553ea9e791" (UID: "0f015abe-55db-486b-a074-43553ea9e791"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:26.909197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.909176    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:26.909197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.909198    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:26.909418 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.909210    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:26.910952 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.910921    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f015abe-55db-486b-a074-43553ea9e791-tls-certs" pod "0f015abe-55db-486b-a074-43553ea9e791" (UID: "0f015abe-55db-486b-a074-43553ea9e791"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:19:26.910952 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.910938    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-dshm" pod "0f015abe-55db-486b-a074-43553ea9e791" (UID: "0f015abe-55db-486b-a074-43553ea9e791"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:26.911156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:26.910957    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f015abe-55db-486b-a074-43553ea9e791-kube-api-access-hnvzx" pod "0f015abe-55db-486b-a074-43553ea9e791" (UID: "0f015abe-55db-486b-a074-43553ea9e791"). InnerVolumeSpecName "kube-api-access-hnvzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:19:27.011173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.011073    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f015abe-55db-486b-a074-43553ea9e791-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:27.011173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.011105    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f015abe-55db-486b-a074-43553ea9e791-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:27.011173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.011121    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnvzx\" (UniqueName: \"kubernetes.io/projected/0f015abe-55db-486b-a074-43553ea9e791-kube-api-access-hnvzx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:27.076856 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.076811    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:27.569217 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.569175    5879 generic.go:356] "Generic (PLEG): container finished" podID="0f015abe-55db-486b-a074-43553ea9e791" containerID="18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4" exitCode=0
Nov 02 13:19:27.569704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.569238    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"
Nov 02 13:19:27.569704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.569259    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg" event={"ID":"0f015abe-55db-486b-a074-43553ea9e791","Type":"ContainerDied","Data":"18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4"}
Nov 02 13:19:27.569704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.569329    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg" event={"ID":"0f015abe-55db-486b-a074-43553ea9e791","Type":"ContainerDied","Data":"9c3b333b921f6f9605237a560cbcedecc84a7c12bca40df7e5beae12ba7a22f6"}
Nov 02 13:19:27.569704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.569354    5879 scope.go:122] "RemoveContainer" containerID="18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4"
Nov 02 13:19:27.585478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.585453    5879 scope.go:122] "RemoveContainer" containerID="18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4"
Nov 02 13:19:27.585841 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:19:27.585813    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4\": container with ID starting with 18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4 not found: ID does not exist" containerID="18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4"
Nov 02 13:19:27.585937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.585854    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4"} err="failed to get container status \"18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4\": rpc error: code = NotFound desc = could not find container \"18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4\": container with ID starting with 18ddbf4d15e649755a4801f8205ccc8a47d40d81b39c47379ca12306bcf3c6a4 not found: ID does not exist"
Nov 02 13:19:27.628086 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.628051    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"]
Nov 02 13:19:27.641720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:27.641685    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bbb7d0/test-model-bbb7d0-kserve-59c45f57df-bkrgg"]
Nov 02 13:19:28.019615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:28.019579    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f015abe-55db-486b-a074-43553ea9e791" path="/var/lib/kubelet/pods/0f015abe-55db-486b-a074-43553ea9e791/volumes"
Nov 02 13:19:28.777088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:28.777042    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:30.568614 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:30.568575    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:32.097276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:32.097243    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:32.223215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:32.223181    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:32.844565 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:32.844531    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:33.591770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.591732    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"]
Nov 02 13:19:33.592818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.592785    5879 cpu_manager.go:414] "containerMap: removing container" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy"
Nov 02 13:19:33.592818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.592813    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy"
Nov 02 13:19:33.593088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.592862    5879 cpu_manager.go:414] "containerMap: removing container" podUID="0f015abe-55db-486b-a074-43553ea9e791" containerName="main"
Nov 02 13:19:33.593088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.592871    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f015abe-55db-486b-a074-43553ea9e791" containerName="main"
Nov 02 13:19:33.593088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.592906    5879 cpu_manager.go:414] "containerMap: removing container" podUID="25175e92-cbb7-46b6-b261-190631c4f98d" containerName="revoke-keys"
Nov 02 13:19:33.593088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.592914    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25175e92-cbb7-46b6-b261-190631c4f98d" containerName="revoke-keys"
Nov 02 13:19:33.593315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.593194    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="31e95cd7-e5a6-446f-90c5-db47c0efe379" containerName="istio-proxy"
Nov 02 13:19:33.593315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.593217    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f015abe-55db-486b-a074-43553ea9e791" containerName="main"
Nov 02 13:19:33.593315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.593233    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="25175e92-cbb7-46b6-b261-190631c4f98d" containerName="revoke-keys"
Nov 02 13:19:33.598051 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.598027    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.621208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.621172    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8962e9-openshift-default-dockercfg-9lgnd\""
Nov 02 13:19:33.643356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.643321    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"]
Nov 02 13:19:33.713140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.713101    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-certs\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.713359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.713159    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw7xw\" (UniqueName: \"kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-kube-api-access-fw7xw\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.713435 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.713407    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-credential-socket\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.713572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.713548    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-data\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.713636 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.713600    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-socket\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.713697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.713675    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-envoy\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.713758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.713737    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942a798-a6e7-427b-a16f-a6ed500bfa61-istiod-ca-cert\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.713828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.713809    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-token\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.713879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.713852    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-podinfo\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.746102 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.746067    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.822772    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-token\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.822837    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-podinfo\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.822963    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-certs\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.823039    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7xw\" (UniqueName: \"kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-kube-api-access-fw7xw\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.823200    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-credential-socket\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.823292    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-data\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.823341    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-socket\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.823387    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-envoy\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.823426    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942a798-a6e7-427b-a16f-a6ed500bfa61-istiod-ca-cert\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.824454    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942a798-a6e7-427b-a16f-a6ed500bfa61-istiod-ca-cert\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.824765    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-credential-socket\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.824959    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-certs\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.825399    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-socket\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.840015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.833534    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-envoy\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.841118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.841061    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-data\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.854292 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.854220    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-podinfo\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.857600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.857562    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7xw\" (UniqueName: \"kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-kube-api-access-fw7xw\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.860791 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.860762    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-token\") pod \"e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") " pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:33.916868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:33.916828    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:34.000829 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:34.000747    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:34.275462 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:19:34.275418    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf942a798_a6e7_427b_a16f_a6ed500bfa61.slice/crio-590d12ec1bff003763c18d774fb5cfbcec59747fd2ef16f67b82e3221544843e WatchSource:0}: Error finding container 590d12ec1bff003763c18d774fb5cfbcec59747fd2ef16f67b82e3221544843e: Status 404 returned error can't find the container with id 590d12ec1bff003763c18d774fb5cfbcec59747fd2ef16f67b82e3221544843e
Nov 02 13:19:34.277236 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:34.277209    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"]
Nov 02 13:19:34.279296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:34.279254    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:19:34.279376 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:34.279347    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:19:34.279418 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:34.279393    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:19:34.653841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:34.653807    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" event={"ID":"f942a798-a6e7-427b-a16f-a6ed500bfa61","Type":"ContainerStarted","Data":"67f967640efc2e64354aa65138ec13065d587fcec2c13d84b3a0b781540f32f0"}
Nov 02 13:19:34.654375 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:34.653861    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" event={"ID":"f942a798-a6e7-427b-a16f-a6ed500bfa61","Type":"ContainerStarted","Data":"590d12ec1bff003763c18d774fb5cfbcec59747fd2ef16f67b82e3221544843e"}
Nov 02 13:19:34.717533 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:34.717474    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" podStartSLOduration=1.717459369 podStartE2EDuration="1.717459369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:19:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:19:34.716819718 +0000 UTC m=+3120.823522569" watchObservedRunningTime="2026-11-02 13:19:34.717459369 +0000 UTC m=+3120.824162219"
Nov 02 13:19:34.918399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:34.918308    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:35.332067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:35.328559    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"]
Nov 02 13:19:35.384328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:35.384297    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:35.801169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:35.801130    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:35.927674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:35.927640    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:35.944275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:35.944205    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:36.688070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:36.688038    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:36.689319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:36.689267    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:19:37.345320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:37.345282    5879 scope.go:122] "RemoveContainer" containerID="b8e98c8044bdebcbbcc04d168d15c50e3d7e227071872a0e9237056a00b34800"
Nov 02 13:19:37.359982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:37.359961    5879 scope.go:122] "RemoveContainer" containerID="d18c81f7c165136af4755cf22fb8d5f2edcec5288724dfc0c045c12df55a2e5f"
Nov 02 13:19:37.371560 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:37.371535    5879 scope.go:122] "RemoveContainer" containerID="5f61951e5143d2de896f5fce23ad77228297abc133c0413f17a5f6b0da578a02"
Nov 02 13:19:37.383949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:37.383926    5879 scope.go:122] "RemoveContainer" containerID="e26f28108c91c5d95f770669b044ad255bc0d3afd6265f9f1d8a62dc02059d7f"
Nov 02 13:19:37.396378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:37.396356    5879 scope.go:122] "RemoveContainer" containerID="e561ed463e1d27651d83564212455963fb1ddd9ab5e14efa832a93259d1b62ff"
Nov 02 13:19:37.407119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:37.407087    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:37.678017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:37.676444    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:39.076307 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.076262    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:39.218967 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.218933    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:39.231453 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.231421    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"]
Nov 02 13:19:39.238942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.238920    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"
Nov 02 13:19:39.257895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.257869    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"]
Nov 02 13:19:39.266602 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.266552    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8962e9\""
Nov 02 13:19:39.266780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.266649    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8962e9-dockercfg-mxblt\""
Nov 02 13:19:39.382940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.373255    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"]
Nov 02 13:19:39.456016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.454749    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"
Nov 02 13:19:39.467448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.466332    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gg5s\" (UniqueName: \"kubernetes.io/projected/1cb1b247-542c-4faa-a654-d6de0ab9b19c-kube-api-access-8gg5s\") pod \"payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc\" (UID: \"1cb1b247-542c-4faa-a654-d6de0ab9b19c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"
Nov 02 13:19:39.467448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.466439    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqhnn\" (UniqueName: \"kubernetes.io/projected/9a8dd290-b517-4712-a174-8e1959783e80-kube-api-access-bqhnn\") pod \"payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z\" (UID: \"9a8dd290-b517-4712-a174-8e1959783e80\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"
Nov 02 13:19:39.467448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.466615    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8dd290-b517-4712-a174-8e1959783e80-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z\" (UID: \"9a8dd290-b517-4712-a174-8e1959783e80\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"
Nov 02 13:19:39.467448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.466863    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb1b247-542c-4faa-a654-d6de0ab9b19c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc\" (UID: \"1cb1b247-542c-4faa-a654-d6de0ab9b19c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"
Nov 02 13:19:39.469166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.469073    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"]
Nov 02 13:19:39.475481 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.475414    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:39.568623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.568581    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb1b247-542c-4faa-a654-d6de0ab9b19c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc\" (UID: \"1cb1b247-542c-4faa-a654-d6de0ab9b19c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"
Nov 02 13:19:39.568817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.568652    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gg5s\" (UniqueName: \"kubernetes.io/projected/1cb1b247-542c-4faa-a654-d6de0ab9b19c-kube-api-access-8gg5s\") pod \"payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc\" (UID: \"1cb1b247-542c-4faa-a654-d6de0ab9b19c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"
Nov 02 13:19:39.568817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.568701    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqhnn\" (UniqueName: \"kubernetes.io/projected/9a8dd290-b517-4712-a174-8e1959783e80-kube-api-access-bqhnn\") pod \"payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z\" (UID: \"9a8dd290-b517-4712-a174-8e1959783e80\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"
Nov 02 13:19:39.568817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.568812    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8dd290-b517-4712-a174-8e1959783e80-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z\" (UID: \"9a8dd290-b517-4712-a174-8e1959783e80\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"
Nov 02 13:19:39.569413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.569393    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb1b247-542c-4faa-a654-d6de0ab9b19c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc\" (UID: \"1cb1b247-542c-4faa-a654-d6de0ab9b19c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"
Nov 02 13:19:39.569542 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.569522    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8dd290-b517-4712-a174-8e1959783e80-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z\" (UID: \"9a8dd290-b517-4712-a174-8e1959783e80\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"
Nov 02 13:19:39.597038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.595202    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gg5s\" (UniqueName: \"kubernetes.io/projected/1cb1b247-542c-4faa-a654-d6de0ab9b19c-kube-api-access-8gg5s\") pod \"payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc\" (UID: \"1cb1b247-542c-4faa-a654-d6de0ab9b19c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"
Nov 02 13:19:39.597038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.596745    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqhnn\" (UniqueName: \"kubernetes.io/projected/9a8dd290-b517-4712-a174-8e1959783e80-kube-api-access-bqhnn\") pod \"payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z\" (UID: \"9a8dd290-b517-4712-a174-8e1959783e80\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"
Nov 02 13:19:39.788159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.788060    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"
Nov 02 13:19:39.856962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.856897    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"
Nov 02 13:19:39.897214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:39.897186    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:40.098442 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:19:40.098411    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8dd290_b517_4712_a174_8e1959783e80.slice/crio-e4f22368b673b1fb7d15c55977a6f21e6e3cf89fde292738a316748b9319fa21 WatchSource:0}: Error finding container e4f22368b673b1fb7d15c55977a6f21e6e3cf89fde292738a316748b9319fa21: Status 404 returned error can't find the container with id e4f22368b673b1fb7d15c55977a6f21e6e3cf89fde292738a316748b9319fa21
Nov 02 13:19:40.100704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:40.100684    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"]
Nov 02 13:19:40.187955 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:19:40.187919    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb1b247_542c_4faa_a654_d6de0ab9b19c.slice/crio-6cb25c139c4786e9aa67867e68a5663aa8be4cccfd557fbb2f6ca58983ef1bb1 WatchSource:0}: Error finding container 6cb25c139c4786e9aa67867e68a5663aa8be4cccfd557fbb2f6ca58983ef1bb1: Status 404 returned error can't find the container with id 6cb25c139c4786e9aa67867e68a5663aa8be4cccfd557fbb2f6ca58983ef1bb1
Nov 02 13:19:40.190140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:40.190041    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"]
Nov 02 13:19:40.746281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:40.746240    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc" event={"ID":"1cb1b247-542c-4faa-a654-d6de0ab9b19c","Type":"ContainerStarted","Data":"6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad"}
Nov 02 13:19:40.746513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:40.746304    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc" event={"ID":"1cb1b247-542c-4faa-a654-d6de0ab9b19c","Type":"ContainerStarted","Data":"6cb25c139c4786e9aa67867e68a5663aa8be4cccfd557fbb2f6ca58983ef1bb1"}
Nov 02 13:19:40.746513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:40.746353    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"
Nov 02 13:19:40.748661 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:40.748632    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z" event={"ID":"9a8dd290-b517-4712-a174-8e1959783e80","Type":"ContainerStarted","Data":"a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5"}
Nov 02 13:19:40.748832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:40.748684    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z" event={"ID":"9a8dd290-b517-4712-a174-8e1959783e80","Type":"ContainerStarted","Data":"e4f22368b673b1fb7d15c55977a6f21e6e3cf89fde292738a316748b9319fa21"}
Nov 02 13:19:40.748832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:40.748710    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"
Nov 02 13:19:40.864506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:40.864453    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc" podStartSLOduration=1.864437821 podStartE2EDuration="1.864437821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:19:40.799655291 +0000 UTC m=+3126.906358142" watchObservedRunningTime="2026-11-02 13:19:40.864437821 +0000 UTC m=+3126.971140691"
Nov 02 13:19:41.283847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:41.283810    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:42.997534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:42.997500    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:43.138522 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:43.138490    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:43.566948 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:43.566911    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:43.724909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:43.724876    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:44.776127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:44.776096    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:44.891335 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:19:44.891271    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:19:46.705372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:46.705340    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:48.495706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:48.495660    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:49.636121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:49.636089    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:49.998452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:49.998361    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:50.354891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.354795    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2" podUID="264eaa3c-030e-46be-a090-502091eed5d5" containerName="main" containerID="cri-o://1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5" gracePeriod=45
Nov 02 13:19:50.665875 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.665844    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:19:50.722139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.722072    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z" podStartSLOduration=11.722052294000001 podStartE2EDuration="11.722052294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:19:40.865133045 +0000 UTC m=+3126.971835895" watchObservedRunningTime="2026-11-02 13:19:50.722052294 +0000 UTC m=+3136.828755147"
Nov 02 13:19:50.759445 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.759412    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-tmp-dir\") pod \"264eaa3c-030e-46be-a090-502091eed5d5\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") "
Nov 02 13:19:50.759632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.759514    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-dshm\") pod \"264eaa3c-030e-46be-a090-502091eed5d5\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") "
Nov 02 13:19:50.759759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.759733    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-tmp-dir" pod "264eaa3c-030e-46be-a090-502091eed5d5" (UID: "264eaa3c-030e-46be-a090-502091eed5d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:50.759847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.759822    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/264eaa3c-030e-46be-a090-502091eed5d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/264eaa3c-030e-46be-a090-502091eed5d5-tls-certs\") pod \"264eaa3c-030e-46be-a090-502091eed5d5\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") "
Nov 02 13:19:50.759903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.759863    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/264eaa3c-030e-46be-a090-502091eed5d5-kube-api-access-jk9vz\" (UniqueName: \"kubernetes.io/projected/264eaa3c-030e-46be-a090-502091eed5d5-kube-api-access-jk9vz\") pod \"264eaa3c-030e-46be-a090-502091eed5d5\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") "
Nov 02 13:19:50.760059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.759987    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-model-cache\") pod \"264eaa3c-030e-46be-a090-502091eed5d5\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") "
Nov 02 13:19:50.760136 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.760083    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-home\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-home\") pod \"264eaa3c-030e-46be-a090-502091eed5d5\" (UID: \"264eaa3c-030e-46be-a090-502091eed5d5\") "
Nov 02 13:19:50.761333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.760496    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-model-cache" pod "264eaa3c-030e-46be-a090-502091eed5d5" (UID: "264eaa3c-030e-46be-a090-502091eed5d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:50.761333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.760716    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-home" pod "264eaa3c-030e-46be-a090-502091eed5d5" (UID: "264eaa3c-030e-46be-a090-502091eed5d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:50.761333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.761263    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:50.761333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.761286    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:50.761333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.761300    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:50.762465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.762439    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/264eaa3c-030e-46be-a090-502091eed5d5-tls-certs" pod "264eaa3c-030e-46be-a090-502091eed5d5" (UID: "264eaa3c-030e-46be-a090-502091eed5d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:19:50.762601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.762506    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-dshm" pod "264eaa3c-030e-46be-a090-502091eed5d5" (UID: "264eaa3c-030e-46be-a090-502091eed5d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:19:50.762666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.762606    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264eaa3c-030e-46be-a090-502091eed5d5-kube-api-access-jk9vz" pod "264eaa3c-030e-46be-a090-502091eed5d5" (UID: "264eaa3c-030e-46be-a090-502091eed5d5"). InnerVolumeSpecName "kube-api-access-jk9vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:19:50.858210 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.858172    5879 generic.go:356] "Generic (PLEG): container finished" podID="264eaa3c-030e-46be-a090-502091eed5d5" containerID="1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5" exitCode=0
Nov 02 13:19:50.858419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.858237    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"
Nov 02 13:19:50.858419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.858258    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2" event={"ID":"264eaa3c-030e-46be-a090-502091eed5d5","Type":"ContainerDied","Data":"1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5"}
Nov 02 13:19:50.858419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.858308    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2" event={"ID":"264eaa3c-030e-46be-a090-502091eed5d5","Type":"ContainerDied","Data":"17d50e9d6fef932d81a6d658a4740cebfb2c98b16fc844beb9300499b75c080f"}
Nov 02 13:19:50.858419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.858325    5879 scope.go:122] "RemoveContainer" containerID="1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5"
Nov 02 13:19:50.862497 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.862475    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/264eaa3c-030e-46be-a090-502091eed5d5-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:50.862617 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.862501    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/264eaa3c-030e-46be-a090-502091eed5d5-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:50.862617 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.862516    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk9vz\" (UniqueName: \"kubernetes.io/projected/264eaa3c-030e-46be-a090-502091eed5d5-kube-api-access-jk9vz\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:19:50.871085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.871065    5879 scope.go:122] "RemoveContainer" containerID="1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5"
Nov 02 13:19:50.871367 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:19:50.871348    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5\": container with ID starting with 1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5 not found: ID does not exist" containerID="1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5"
Nov 02 13:19:50.871425 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.871374    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5"} err="failed to get container status \"1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5\": rpc error: code = NotFound desc = could not find container \"1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5\": container with ID starting with 1ef4f9da9b90af43de56584484c866cfc6efb3553d52430ef5cdd9e251ba03c5 not found: ID does not exist"
Nov 02 13:19:50.917970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.917893    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"]
Nov 02 13:19:50.921747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:50.921717    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9139ec/test-model-9139ec-kserve-7ff57b865f-mv8k2"]
Nov 02 13:19:51.698626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:51.698587    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:51.760785 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:51.760755    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"
Nov 02 13:19:51.761026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:51.760840    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"
Nov 02 13:19:52.019251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:52.019176    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="264eaa3c-030e-46be-a090-502091eed5d5" path="/var/lib/kubelet/pods/264eaa3c-030e-46be-a090-502091eed5d5/volumes"
Nov 02 13:19:53.438801 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:53.438763    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:54.973959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:54.973926    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:56.535330 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:56.535295    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:58.194770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:58.194740    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:19:58.336287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:19:58.336256    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:01.255691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:01.255658    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:01.557424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:01.557343    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:03.226013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:03.225963    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:04.758529 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:04.758488    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:06.367263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:06.367224    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:07.882662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:07.882627    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:09.539250 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:09.539214    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:11.152321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:11.152279    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:12.359074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:12.359040    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:12.779510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:12.779474    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:14.444210 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:14.444132    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:14.823169 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:20:14.823071    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:20:15.991745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:15.991711    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:17.584428 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:17.584391    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:19.241223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:19.241190    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:20.878466 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:20.878430    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:22.452750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:22.452677    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:23.817536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:23.817501    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:24.545849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:24.545804    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:26.219648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:26.219589    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:28.182862 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:28.182828    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:29.866147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:29.866112    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:31.526819 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:31.526788    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:31.681274 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:31.681229    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:33.483703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:33.483671    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:34.011427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:34.011393    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:34.180961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:34.180927    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:34.815366 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:34.815326    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:35.236878 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:35.236841    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:35.767700 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:35.767665    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:36.074390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:36.074303    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:37.502441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:37.502407    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:39.251552 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:39.251503    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:40.017796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:40.017767    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:40.807121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:40.807085    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:41.149404 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:41.149369    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"]
Nov 02 13:20:41.185292 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:41.185259    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"]
Nov 02 13:20:41.372566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:41.371361    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:41.496081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:41.495796    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:41.632640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:41.632599    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:42.671698 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:42.671660    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:42.803396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:42.803366    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:44.270744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:44.270704    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:44.823935 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:20:44.823893    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:20:45.932176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:45.932133    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:46.150090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:46.150042    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6" podUID="7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a" containerName="payload-processing" containerID="cri-o://0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49" gracePeriod=25
Nov 02 13:20:46.186341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:46.186255    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz" podUID="abe4af2d-1652-47e4-8883-0ca27eb781db" containerName="payload-pre-processing" containerID="cri-o://fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd" gracePeriod=25
Nov 02 13:20:47.569484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:47.569442    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:48.828580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:48.828544    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:49.143481 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:49.143434    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:50.717586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:50.717554    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:51.773888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.773855    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"]
Nov 02 13:20:51.774720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.774704    5879 cpu_manager.go:414] "containerMap: removing container" podUID="264eaa3c-030e-46be-a090-502091eed5d5" containerName="main"
Nov 02 13:20:51.774720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.774720    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="264eaa3c-030e-46be-a090-502091eed5d5" containerName="main"
Nov 02 13:20:51.774892 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.774882    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="264eaa3c-030e-46be-a090-502091eed5d5" containerName="main"
Nov 02 13:20:51.780195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.780174    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:51.800584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.800552    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"]
Nov 02 13:20:51.809341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.809306    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:20:51.809671 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.809643    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:20:51.856938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.856899    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19751a1d-pnk6q\" (UID: \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:51.857144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.857057    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vfv\" (UniqueName: \"kubernetes.io/projected/574d8033-d2ce-4b22-b3c0-a8ac82d989da-kube-api-access-v4vfv\") pod \"maas-api-revoke-sub-19751a1d-pnk6q\" (UID: \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:51.857205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.857183    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19751a1d-pnk6q\" (UID: \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:51.958822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.958789    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19751a1d-pnk6q\" (UID: \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:51.959065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.959047    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19751a1d-pnk6q\" (UID: \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:51.959163 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.959147    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vfv\" (UniqueName: \"kubernetes.io/projected/574d8033-d2ce-4b22-b3c0-a8ac82d989da-kube-api-access-v4vfv\") pod \"maas-api-revoke-sub-19751a1d-pnk6q\" (UID: \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:51.959714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.959695    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19751a1d-pnk6q\" (UID: \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:51.961434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.961415    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19751a1d-pnk6q\" (UID: \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:51.984600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:51.984574    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vfv\" (UniqueName: \"kubernetes.io/projected/574d8033-d2ce-4b22-b3c0-a8ac82d989da-kube-api-access-v4vfv\") pod \"maas-api-revoke-sub-19751a1d-pnk6q\" (UID: \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:52.095438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:52.095359    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:52.230195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:52.230143    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz" podUID="abe4af2d-1652-47e4-8883-0ca27eb781db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:20:52.230423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:52.230206    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6" podUID="7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:20:52.342151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:52.342118    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:52.412232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:52.412196    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"]
Nov 02 13:20:52.415564 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:20:52.415522    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574d8033_d2ce_4b22_b3c0_a8ac82d989da.slice/crio-9749853a83c4787f467624828b7747705d0677b5afb24fb1816c6ea7ebddfb81 WatchSource:0}: Error finding container 9749853a83c4787f467624828b7747705d0677b5afb24fb1816c6ea7ebddfb81: Status 404 returned error can't find the container with id 9749853a83c4787f467624828b7747705d0677b5afb24fb1816c6ea7ebddfb81
Nov 02 13:20:53.425927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:53.425784    5879 generic.go:356] "Generic (PLEG): container finished" podID="574d8033-d2ce-4b22-b3c0-a8ac82d989da" containerID="93977a7222d62e3200fd8aca70b74ea5991970a009f55b6b341fb3771074ee71" exitCode=0
Nov 02 13:20:53.425927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:53.425914    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q" event={"ID":"574d8033-d2ce-4b22-b3c0-a8ac82d989da","Type":"ContainerDied","Data":"93977a7222d62e3200fd8aca70b74ea5991970a009f55b6b341fb3771074ee71"}
Nov 02 13:20:53.426421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:53.425976    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q" event={"ID":"574d8033-d2ce-4b22-b3c0-a8ac82d989da","Type":"ContainerStarted","Data":"9749853a83c4787f467624828b7747705d0677b5afb24fb1816c6ea7ebddfb81"}
Nov 02 13:20:53.946198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:53.946162    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:54.721503 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:54.721476    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:54.836838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:54.836808    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/574d8033-d2ce-4b22-b3c0-a8ac82d989da-kube-api-access-v4vfv\" (UniqueName: \"kubernetes.io/projected/574d8033-d2ce-4b22-b3c0-a8ac82d989da-kube-api-access-v4vfv\") pod \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\" (UID: \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\") "
Nov 02 13:20:54.837089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:54.836875    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-openshift-ca\") pod \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\" (UID: \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\") "
Nov 02 13:20:54.837089 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:54.837032    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-xks-ca\") pod \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\" (UID: \"574d8033-d2ce-4b22-b3c0-a8ac82d989da\") "
Nov 02 13:20:54.837442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:54.837418    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-openshift-ca" pod "574d8033-d2ce-4b22-b3c0-a8ac82d989da" (UID: "574d8033-d2ce-4b22-b3c0-a8ac82d989da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:20:54.837760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:54.837738    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:20:54.839249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:54.839227    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574d8033-d2ce-4b22-b3c0-a8ac82d989da-kube-api-access-v4vfv" pod "574d8033-d2ce-4b22-b3c0-a8ac82d989da" (UID: "574d8033-d2ce-4b22-b3c0-a8ac82d989da"). InnerVolumeSpecName "kube-api-access-v4vfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:20:54.839336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:54.839264    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-xks-ca" pod "574d8033-d2ce-4b22-b3c0-a8ac82d989da" (UID: "574d8033-d2ce-4b22-b3c0-a8ac82d989da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:20:54.939974 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:54.939940    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574d8033-d2ce-4b22-b3c0-a8ac82d989da-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:20:54.939974 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:54.939975    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4vfv\" (UniqueName: \"kubernetes.io/projected/574d8033-d2ce-4b22-b3c0-a8ac82d989da-kube-api-access-v4vfv\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:20:55.471047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:55.466946    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q" event={"ID":"574d8033-d2ce-4b22-b3c0-a8ac82d989da","Type":"ContainerDied","Data":"9749853a83c4787f467624828b7747705d0677b5afb24fb1816c6ea7ebddfb81"}
Nov 02 13:20:55.471047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:55.467029    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9749853a83c4787f467624828b7747705d0677b5afb24fb1816c6ea7ebddfb81"
Nov 02 13:20:55.471047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:55.467092    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"
Nov 02 13:20:55.642397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:55.642365    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:57.607016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:57.601369    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:20:59.321699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:20:59.321649    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:00.844388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:00.844348    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:01.104096 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:01.103433    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:02.229853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.229814    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz" podUID="abe4af2d-1652-47e4-8883-0ca27eb781db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:21:02.229853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.229846    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6" podUID="7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:21:02.607035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.606930    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"]
Nov 02 13:21:02.608107 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.608082    5879 cpu_manager.go:414] "containerMap: removing container" podUID="574d8033-d2ce-4b22-b3c0-a8ac82d989da" containerName="revoke-keys"
Nov 02 13:21:02.608107 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.608107    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="574d8033-d2ce-4b22-b3c0-a8ac82d989da" containerName="revoke-keys"
Nov 02 13:21:02.608985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.608950    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="574d8033-d2ce-4b22-b3c0-a8ac82d989da" containerName="revoke-keys"
Nov 02 13:21:02.621505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.621478    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.643937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.643701    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e8a402\"/\"test-model-e8a402-kserve-self-signed-certs\""
Nov 02 13:21:02.644192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.644169    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e8a402\"/\"openshift-service-ca.crt\""
Nov 02 13:21:02.644796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.644771    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e8a402\"/\"kube-root-ca.crt\""
Nov 02 13:21:02.645463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.645435    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"]
Nov 02 13:21:02.661048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.659333    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e8a402\"/\"default-dockercfg-2w48d\""
Nov 02 13:21:02.690970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.690931    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tmp-dir\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.691530 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.691506    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-745md\" (UniqueName: \"kubernetes.io/projected/dfcead4b-186c-4c78-8d1d-93d07d3c450a-kube-api-access-745md\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.691820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.691740    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-dshm\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.692279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.692161    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-model-cache\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.692483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.692455    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tls-certs\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.692683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.692667    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-home\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.794202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.794162    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-model-cache\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.794426 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.794222    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tls-certs\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.794426 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.794270    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-home\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.794426 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.794369    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tmp-dir\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.794426 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.794418    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-745md\" (UniqueName: \"kubernetes.io/projected/dfcead4b-186c-4c78-8d1d-93d07d3c450a-kube-api-access-745md\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.794660 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.794487    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-dshm\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.794720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.794682    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-model-cache\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.794787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.794765    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tmp-dir\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.795058 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.795023    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-home\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.797162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.797139    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tls-certs\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.797506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.797486    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-dshm\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.814940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.814902    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:02.820540 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.820508    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-745md\" (UniqueName: \"kubernetes.io/projected/dfcead4b-186c-4c78-8d1d-93d07d3c450a-kube-api-access-745md\") pod \"test-model-e8a402-kserve-85d666df87-sz2bk\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") " pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.964276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.963972    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:02.966357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:02.966325    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:03.333014 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:21:03.332964    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfcead4b_186c_4c78_8d1d_93d07d3c450a.slice/crio-e4b8c44f0df0eb3f36cc191ccba5eec2c5e3ce3c21728ce97efa77d1f8140047 WatchSource:0}: Error finding container e4b8c44f0df0eb3f36cc191ccba5eec2c5e3ce3c21728ce97efa77d1f8140047: Status 404 returned error can't find the container with id e4b8c44f0df0eb3f36cc191ccba5eec2c5e3ce3c21728ce97efa77d1f8140047
Nov 02 13:21:03.335674 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:03.335543    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"]
Nov 02 13:21:03.565472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:03.565398    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk" event={"ID":"dfcead4b-186c-4c78-8d1d-93d07d3c450a","Type":"ContainerStarted","Data":"ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313"}
Nov 02 13:21:03.565660 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:03.565486    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk" event={"ID":"dfcead4b-186c-4c78-8d1d-93d07d3c450a","Type":"ContainerStarted","Data":"e4b8c44f0df0eb3f36cc191ccba5eec2c5e3ce3c21728ce97efa77d1f8140047"}
Nov 02 13:21:03.613485 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:03.613373    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk" podStartSLOduration=1.6133528209999999 podStartE2EDuration="1.613352821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:21:03.6104038 +0000 UTC m=+3209.717106652" watchObservedRunningTime="2026-11-02 13:21:03.613352821 +0000 UTC m=+3209.720055671"
Nov 02 13:21:03.965172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:03.965077    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:04.630764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:04.630725    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:06.297429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.297403    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:06.572971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.572649    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"
Nov 02 13:21:06.586390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.586358    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"
Nov 02 13:21:06.594261 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.594232    5879 generic.go:356] "Generic (PLEG): container finished" podID="abe4af2d-1652-47e4-8883-0ca27eb781db" containerID="fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd" exitCode=0
Nov 02 13:21:06.594464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.594277    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"
Nov 02 13:21:06.594464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.594319    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz" event={"ID":"abe4af2d-1652-47e4-8883-0ca27eb781db","Type":"ContainerDied","Data":"fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd"}
Nov 02 13:21:06.594464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.594415    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz" event={"ID":"abe4af2d-1652-47e4-8883-0ca27eb781db","Type":"ContainerDied","Data":"8e4a41f777ec6dda194a9fd86f6294c16ec7cd724c37ec4caed2640aada539dc"}
Nov 02 13:21:06.594464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.594445    5879 scope.go:122] "RemoveContainer" containerID="fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd"
Nov 02 13:21:06.595872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.595837    5879 generic.go:356] "Generic (PLEG): container finished" podID="7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a" containerID="0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49" exitCode=0
Nov 02 13:21:06.596043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.595876    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"
Nov 02 13:21:06.596043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.595919    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6" event={"ID":"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a","Type":"ContainerDied","Data":"0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49"}
Nov 02 13:21:06.596043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.595966    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6" event={"ID":"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a","Type":"ContainerDied","Data":"cf54e928710891ecf80a217eff72f984854f9c1e84fc2eff48a35525d1186260"}
Nov 02 13:21:06.598189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.598165    5879 generic.go:356] "Generic (PLEG): container finished" podID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerID="67f967640efc2e64354aa65138ec13065d587fcec2c13d84b3a0b781540f32f0" exitCode=0
Nov 02 13:21:06.598321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.598206    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" event={"ID":"f942a798-a6e7-427b-a16f-a6ed500bfa61","Type":"ContainerDied","Data":"67f967640efc2e64354aa65138ec13065d587fcec2c13d84b3a0b781540f32f0"}
Nov 02 13:21:06.598648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.598634    5879 scope.go:122] "RemoveContainer" containerID="67f967640efc2e64354aa65138ec13065d587fcec2c13d84b3a0b781540f32f0"
Nov 02 13:21:06.600883 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.600845    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:21:06.600978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.600943    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:21:06.601056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.601039    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:21:06.606841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.606823    5879 scope.go:122] "RemoveContainer" containerID="fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd"
Nov 02 13:21:06.607157 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:21:06.607137    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd\": container with ID starting with fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd not found: ID does not exist" containerID="fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd"
Nov 02 13:21:06.607223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.607167    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd"} err="failed to get container status \"fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd\": rpc error: code = NotFound desc = could not find container \"fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd\": container with ID starting with fd613762760a2c8cc9c28023f7400484bd103358469c1427c6aa2d1de79619bd not found: ID does not exist"
Nov 02 13:21:06.607223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.607185    5879 scope.go:122] "RemoveContainer" containerID="0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49"
Nov 02 13:21:06.624265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.624238    5879 scope.go:122] "RemoveContainer" containerID="0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49"
Nov 02 13:21:06.624671 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:21:06.624643    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49\": container with ID starting with 0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49 not found: ID does not exist" containerID="0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49"
Nov 02 13:21:06.624764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.624683    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49"} err="failed to get container status \"0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49\": rpc error: code = NotFound desc = could not find container \"0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49\": container with ID starting with 0b460ed0458b6e08cc2d6e35a368464157f6f080b773cd8d5090e3d5fe3d3c49 not found: ID does not exist"
Nov 02 13:21:06.667372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.667279    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abe4af2d-1652-47e4-8883-0ca27eb781db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe4af2d-1652-47e4-8883-0ca27eb781db-plugins-config-volume\") pod \"abe4af2d-1652-47e4-8883-0ca27eb781db\" (UID: \"abe4af2d-1652-47e4-8883-0ca27eb781db\") "
Nov 02 13:21:06.669140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.667416    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-kube-api-access-l6d6z\" (UniqueName: \"kubernetes.io/projected/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-kube-api-access-l6d6z\") pod \"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a\" (UID: \"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a\") "
Nov 02 13:21:06.669140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.667451    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-plugins-config-volume\") pod \"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a\" (UID: \"7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a\") "
Nov 02 13:21:06.669140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.667603    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abe4af2d-1652-47e4-8883-0ca27eb781db-kube-api-access-vshdk\" (UniqueName: \"kubernetes.io/projected/abe4af2d-1652-47e4-8883-0ca27eb781db-kube-api-access-vshdk\") pod \"abe4af2d-1652-47e4-8883-0ca27eb781db\" (UID: \"abe4af2d-1652-47e4-8883-0ca27eb781db\") "
Nov 02 13:21:06.669140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.667742    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abe4af2d-1652-47e4-8883-0ca27eb781db-plugins-config-volume" pod "abe4af2d-1652-47e4-8883-0ca27eb781db" (UID: "abe4af2d-1652-47e4-8883-0ca27eb781db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:21:06.669140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.668127    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-plugins-config-volume" pod "7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a" (UID: "7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:21:06.669140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.668774    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abe4af2d-1652-47e4-8883-0ca27eb781db-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:06.669140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.668797    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:06.671825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.671791    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-kube-api-access-l6d6z" pod "7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a" (UID: "7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a"). InnerVolumeSpecName "kube-api-access-l6d6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:06.673682 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.673653    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe4af2d-1652-47e4-8883-0ca27eb781db-kube-api-access-vshdk" pod "abe4af2d-1652-47e4-8883-0ca27eb781db" (UID: "abe4af2d-1652-47e4-8883-0ca27eb781db"). InnerVolumeSpecName "kube-api-access-vshdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:06.688653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.688627    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:21:06.770885 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.770850    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vshdk\" (UniqueName: \"kubernetes.io/projected/abe4af2d-1652-47e4-8883-0ca27eb781db-kube-api-access-vshdk\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:06.770885 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.770882    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6d6z\" (UniqueName: \"kubernetes.io/projected/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a-kube-api-access-l6d6z\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:06.949809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.949728    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"]
Nov 02 13:21:06.953834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.953804    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9139ec-75499f56ff-2zqj6"]
Nov 02 13:21:06.994082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.994040    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"]
Nov 02 13:21:06.998231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:06.998200    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9139ec-d598b575f-8w9zz"]
Nov 02 13:21:07.626948 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:07.626912    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" event={"ID":"f942a798-a6e7-427b-a16f-a6ed500bfa61","Type":"ContainerStarted","Data":"d4ea8c0870cb2c9ee6524121127a3ba3eab870238c7f75bc51110f0abd1d4871"}
Nov 02 13:21:07.918019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:07.917910    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:21:07.919222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:07.919196    5879 patch_prober.go:29] interesting pod/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body=
Nov 02 13:21:07.919358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:07.919253    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused"
Nov 02 13:21:08.019793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:08.019755    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a" path="/var/lib/kubelet/pods/7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a/volumes"
Nov 02 13:21:08.020297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:08.020282    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abe4af2d-1652-47e4-8883-0ca27eb781db" path="/var/lib/kubelet/pods/abe4af2d-1652-47e4-8883-0ca27eb781db/volumes"
Nov 02 13:21:08.045512 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:08.045464    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:08.918808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:08.918764    5879 patch_prober.go:29] interesting pod/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body=
Nov 02 13:21:08.919244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:08.918832    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused"
Nov 02 13:21:09.746720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:09.746686    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:09.918844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:09.918804    5879 patch_prober.go:29] interesting pod/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body=
Nov 02 13:21:09.919338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:09.918870    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused"
Nov 02 13:21:10.918300 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:10.918265    5879 patch_prober.go:29] interesting pod/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body=
Nov 02 13:21:10.918517 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:10.918322    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused"
Nov 02 13:21:11.462789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:11.462758    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:11.918821 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:11.918787    5879 patch_prober.go:29] interesting pod/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body=
Nov 02 13:21:11.919033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:11.918841    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused"
Nov 02 13:21:12.918353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:12.918309    5879 patch_prober.go:29] interesting pod/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body=
Nov 02 13:21:12.918864 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:12.918367    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused"
Nov 02 13:21:12.965687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:12.965645    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:12.978954 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:12.978925    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:13.158069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:13.158036    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:13.691457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:13.691428    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:21:13.924123 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:13.924091    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:21:13.924595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:13.924517    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:21:13.925079 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:13.925063    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:21:14.823490 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:21:14.823443    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:21:14.849731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:14.849702    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:16.506968 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:16.506934    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:18.164487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.164454    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:18.336354 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.336322    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"]
Nov 02 13:21:18.385739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.385702    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:18.851270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.851235    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"]
Nov 02 13:21:18.852133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.852115    5879 cpu_manager.go:414] "containerMap: removing container" podUID="abe4af2d-1652-47e4-8883-0ca27eb781db" containerName="payload-pre-processing"
Nov 02 13:21:18.852220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.852134    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abe4af2d-1652-47e4-8883-0ca27eb781db" containerName="payload-pre-processing"
Nov 02 13:21:18.852220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.852157    5879 cpu_manager.go:414] "containerMap: removing container" podUID="7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a" containerName="payload-processing"
Nov 02 13:21:18.852220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.852163    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a" containerName="payload-processing"
Nov 02 13:21:18.852326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.852319    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc45a3d-f7bf-43a3-a0a1-daa8bd1bb89a" containerName="payload-processing"
Nov 02 13:21:18.852379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.852340    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="abe4af2d-1652-47e4-8883-0ca27eb781db" containerName="payload-pre-processing"
Nov 02 13:21:18.856086 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.856067    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:18.872544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.872518    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:21:18.878065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.878024    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"]
Nov 02 13:21:18.886416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:18.886391    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:21:19.016209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.016168    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e49734de-tjmdx\" (UID: \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:19.016421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.016301    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck56t\" (UniqueName: \"kubernetes.io/projected/21f2c3b6-4105-48d2-a8ae-33df7141fc13-kube-api-access-ck56t\") pod \"maas-api-revoke-sub-e49734de-tjmdx\" (UID: \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:19.016498 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.016481    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e49734de-tjmdx\" (UID: \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:19.118307 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.118275    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e49734de-tjmdx\" (UID: \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:19.118480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.118394    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck56t\" (UniqueName: \"kubernetes.io/projected/21f2c3b6-4105-48d2-a8ae-33df7141fc13-kube-api-access-ck56t\") pod \"maas-api-revoke-sub-e49734de-tjmdx\" (UID: \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:19.118670 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.118648    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e49734de-tjmdx\" (UID: \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:19.119309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.119285    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e49734de-tjmdx\" (UID: \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:19.121008 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.120969    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e49734de-tjmdx\" (UID: \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:19.144106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.144080    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck56t\" (UniqueName: \"kubernetes.io/projected/21f2c3b6-4105-48d2-a8ae-33df7141fc13-kube-api-access-ck56t\") pod \"maas-api-revoke-sub-e49734de-tjmdx\" (UID: \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:19.171196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.171171    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:19.483453 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:21:19.483414    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f2c3b6_4105_48d2_a8ae_33df7141fc13.slice/crio-13621d05c746aad72c5e5e2e558b81ece471330464ccea4b82583b620510c4d6 WatchSource:0}: Error finding container 13621d05c746aad72c5e5e2e558b81ece471330464ccea4b82583b620510c4d6: Status 404 returned error can't find the container with id 13621d05c746aad72c5e5e2e558b81ece471330464ccea4b82583b620510c4d6
Nov 02 13:21:19.485489 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.485455    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"]
Nov 02 13:21:19.731205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.731117    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx" event={"ID":"21f2c3b6-4105-48d2-a8ae-33df7141fc13","Type":"ContainerStarted","Data":"13621d05c746aad72c5e5e2e558b81ece471330464ccea4b82583b620510c4d6"}
Nov 02 13:21:19.839026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:19.838959    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:20.741259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:20.741219    5879 generic.go:356] "Generic (PLEG): container finished" podID="21f2c3b6-4105-48d2-a8ae-33df7141fc13" containerID="28b2663860d3efb7c385c53fb1b0f4a09dcf3e05b6214aefda7d878a65a88ba0" exitCode=0
Nov 02 13:21:20.741673 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:20.741301    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx" event={"ID":"21f2c3b6-4105-48d2-a8ae-33df7141fc13","Type":"ContainerDied","Data":"28b2663860d3efb7c385c53fb1b0f4a09dcf3e05b6214aefda7d878a65a88ba0"}
Nov 02 13:21:21.431925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.431887    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:21.660672 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.660621    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"]
Nov 02 13:21:21.664660 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.664637    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:21.688439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.688353    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"]
Nov 02 13:21:21.762554 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.762520    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d62db4d8-thcb6\" (UID: \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:21.763110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.762809    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c7qp\" (UniqueName: \"kubernetes.io/projected/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-kube-api-access-6c7qp\") pod \"maas-api-revoke-sub-d62db4d8-thcb6\" (UID: \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:21.763110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.762917    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d62db4d8-thcb6\" (UID: \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:21.865304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.865250    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d62db4d8-thcb6\" (UID: \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:21.865516 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.865472    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d62db4d8-thcb6\" (UID: \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:21.865747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.865727    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c7qp\" (UniqueName: \"kubernetes.io/projected/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-kube-api-access-6c7qp\") pod \"maas-api-revoke-sub-d62db4d8-thcb6\" (UID: \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:21.866491 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.866461    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d62db4d8-thcb6\" (UID: \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:21.869423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.869387    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d62db4d8-thcb6\" (UID: \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:21.891306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.891275    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c7qp\" (UniqueName: \"kubernetes.io/projected/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-kube-api-access-6c7qp\") pod \"maas-api-revoke-sub-d62db4d8-thcb6\" (UID: \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:21.951273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.951200    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:21.979473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:21.979446    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:22.072045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.068477    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-xks-ca\") pod \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\" (UID: \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\") "
Nov 02 13:21:22.072045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.068549    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f2c3b6-4105-48d2-a8ae-33df7141fc13-kube-api-access-ck56t\" (UniqueName: \"kubernetes.io/projected/21f2c3b6-4105-48d2-a8ae-33df7141fc13-kube-api-access-ck56t\") pod \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\" (UID: \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\") "
Nov 02 13:21:22.072045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.068796    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-openshift-ca\") pod \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\" (UID: \"21f2c3b6-4105-48d2-a8ae-33df7141fc13\") "
Nov 02 13:21:22.072045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.070558    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-openshift-ca" pod "21f2c3b6-4105-48d2-a8ae-33df7141fc13" (UID: "21f2c3b6-4105-48d2-a8ae-33df7141fc13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:21:22.074057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.073115    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-xks-ca" pod "21f2c3b6-4105-48d2-a8ae-33df7141fc13" (UID: "21f2c3b6-4105-48d2-a8ae-33df7141fc13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:21:22.083149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.083115    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f2c3b6-4105-48d2-a8ae-33df7141fc13-kube-api-access-ck56t" pod "21f2c3b6-4105-48d2-a8ae-33df7141fc13" (UID: "21f2c3b6-4105-48d2-a8ae-33df7141fc13"). InnerVolumeSpecName "kube-api-access-ck56t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:22.172214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.172176    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:22.172214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.172214    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck56t\" (UniqueName: \"kubernetes.io/projected/21f2c3b6-4105-48d2-a8ae-33df7141fc13-kube-api-access-ck56t\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:22.172430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.172231    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21f2c3b6-4105-48d2-a8ae-33df7141fc13-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:22.304407 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:21:22.304377    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142cb7a8_c0b0_40e6_97f9_5071c87ad3fd.slice/crio-a9c366960f7e0aab81a16df52ddda9141c0b5ea09d7f840fd9bf6df953128da1 WatchSource:0}: Error finding container a9c366960f7e0aab81a16df52ddda9141c0b5ea09d7f840fd9bf6df953128da1: Status 404 returned error can't find the container with id a9c366960f7e0aab81a16df52ddda9141c0b5ea09d7f840fd9bf6df953128da1
Nov 02 13:21:22.306812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.306782    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"]
Nov 02 13:21:22.760108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.760067    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6" event={"ID":"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd","Type":"ContainerStarted","Data":"a9c366960f7e0aab81a16df52ddda9141c0b5ea09d7f840fd9bf6df953128da1"}
Nov 02 13:21:22.761655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.761628    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx" event={"ID":"21f2c3b6-4105-48d2-a8ae-33df7141fc13","Type":"ContainerDied","Data":"13621d05c746aad72c5e5e2e558b81ece471330464ccea4b82583b620510c4d6"}
Nov 02 13:21:22.761802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.761671    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"
Nov 02 13:21:22.761802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:22.761677    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13621d05c746aad72c5e5e2e558b81ece471330464ccea4b82583b620510c4d6"
Nov 02 13:21:23.130271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:23.130237    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:23.772817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:23.772736    5879 generic.go:356] "Generic (PLEG): container finished" podID="142cb7a8-c0b0-40e6-97f9-5071c87ad3fd" containerID="efd24be06fb45985544760c8ec48a3123254794da7f2500d738314d6dc70203c" exitCode=0
Nov 02 13:21:23.772984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:23.772829    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6" event={"ID":"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd","Type":"ContainerDied","Data":"efd24be06fb45985544760c8ec48a3123254794da7f2500d738314d6dc70203c"}
Nov 02 13:21:24.871313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:24.871255    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:24.970632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:24.970603    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:25.018040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.017987    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-xks-ca\") pod \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\" (UID: \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\") "
Nov 02 13:21:25.018294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.018162    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-kube-api-access-6c7qp\" (UniqueName: \"kubernetes.io/projected/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-kube-api-access-6c7qp\") pod \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\" (UID: \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\") "
Nov 02 13:21:25.018367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.018338    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-openshift-ca\") pod \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\" (UID: \"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd\") "
Nov 02 13:21:25.019624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.019596    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-openshift-ca" pod "142cb7a8-c0b0-40e6-97f9-5071c87ad3fd" (UID: "142cb7a8-c0b0-40e6-97f9-5071c87ad3fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:21:25.020626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.020603    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-xks-ca" pod "142cb7a8-c0b0-40e6-97f9-5071c87ad3fd" (UID: "142cb7a8-c0b0-40e6-97f9-5071c87ad3fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:21:25.021823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.021795    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-kube-api-access-6c7qp" pod "142cb7a8-c0b0-40e6-97f9-5071c87ad3fd" (UID: "142cb7a8-c0b0-40e6-97f9-5071c87ad3fd"). InnerVolumeSpecName "kube-api-access-6c7qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:25.120484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.120442    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c7qp\" (UniqueName: \"kubernetes.io/projected/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-kube-api-access-6c7qp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:25.120484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.120472    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:25.120484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.120486    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:25.793294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.793259    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"
Nov 02 13:21:25.793506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.793292    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6" event={"ID":"142cb7a8-c0b0-40e6-97f9-5071c87ad3fd","Type":"ContainerDied","Data":"a9c366960f7e0aab81a16df52ddda9141c0b5ea09d7f840fd9bf6df953128da1"}
Nov 02 13:21:25.793506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.793345    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9c366960f7e0aab81a16df52ddda9141c0b5ea09d7f840fd9bf6df953128da1"
Nov 02 13:21:25.927796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.927760    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"]
Nov 02 13:21:25.928632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.928615    5879 cpu_manager.go:414] "containerMap: removing container" podUID="142cb7a8-c0b0-40e6-97f9-5071c87ad3fd" containerName="revoke-keys"
Nov 02 13:21:25.928632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.928633    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="142cb7a8-c0b0-40e6-97f9-5071c87ad3fd" containerName="revoke-keys"
Nov 02 13:21:25.928768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.928670    5879 cpu_manager.go:414] "containerMap: removing container" podUID="21f2c3b6-4105-48d2-a8ae-33df7141fc13" containerName="revoke-keys"
Nov 02 13:21:25.928768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.928675    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f2c3b6-4105-48d2-a8ae-33df7141fc13" containerName="revoke-keys"
Nov 02 13:21:25.928841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.928804    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f2c3b6-4105-48d2-a8ae-33df7141fc13" containerName="revoke-keys"
Nov 02 13:21:25.928841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.928822    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="142cb7a8-c0b0-40e6-97f9-5071c87ad3fd" containerName="revoke-keys"
Nov 02 13:21:25.932147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.932115    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:25.950832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.949810    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-357b30\"/\"kube-root-ca.crt\""
Nov 02 13:21:25.950832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.949893    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-357b30\"/\"openshift-service-ca.crt\""
Nov 02 13:21:25.963537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.963510    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-357b30\"/\"test-model-357b30-kserve-self-signed-certs\""
Nov 02 13:21:25.963537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.963530    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-357b30\"/\"default-dockercfg-2nb98\""
Nov 02 13:21:25.970957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:25.970928    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"]
Nov 02 13:21:26.033959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.033924    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-home\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.034179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.033971    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-dshm\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.034179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.034154    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-model-cache\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.034281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.034209    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-tmp-dir\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.034420 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.034405    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa69de3-1d00-44f8-af1e-796146e70910-tls-certs\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.034542 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.034528    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpjnq\" (UniqueName: \"kubernetes.io/projected/4aa69de3-1d00-44f8-af1e-796146e70910-kube-api-access-lpjnq\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.136528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.136487    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa69de3-1d00-44f8-af1e-796146e70910-tls-certs\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.136736 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.136639    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpjnq\" (UniqueName: \"kubernetes.io/projected/4aa69de3-1d00-44f8-af1e-796146e70910-kube-api-access-lpjnq\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.136840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.136751    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-home\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.136840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.136783    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-dshm\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.136977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.136890    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-model-cache\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.136977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.136936    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-tmp-dir\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.137326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.137297    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-model-cache\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.137507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.137351    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-tmp-dir\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.137905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.137878    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-home\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.139484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.139463    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa69de3-1d00-44f8-af1e-796146e70910-tls-certs\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.140101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.140084    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-dshm\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.167015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.166158    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpjnq\" (UniqueName: \"kubernetes.io/projected/4aa69de3-1d00-44f8-af1e-796146e70910-kube-api-access-lpjnq\") pod \"test-model-357b30-kserve-69d478754c-5g5kq\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") " pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.255057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.253695    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:26.616955 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:21:26.616922    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4aa69de3_1d00_44f8_af1e_796146e70910.slice/crio-de8d947b57d76c3c21840c240c523e7fd950b18de17efe3c50b6dd45b3f8ea26 WatchSource:0}: Error finding container de8d947b57d76c3c21840c240c523e7fd950b18de17efe3c50b6dd45b3f8ea26: Status 404 returned error can't find the container with id de8d947b57d76c3c21840c240c523e7fd950b18de17efe3c50b6dd45b3f8ea26
Nov 02 13:21:26.618712 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.618689    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"]
Nov 02 13:21:26.804085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.804047    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq" event={"ID":"4aa69de3-1d00-44f8-af1e-796146e70910","Type":"ContainerStarted","Data":"f80bd6e5ff469447517e76f814de3bb949db5b95d99f30885a7742930053aa04"}
Nov 02 13:21:26.804253 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.804099    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq" event={"ID":"4aa69de3-1d00-44f8-af1e-796146e70910","Type":"ContainerStarted","Data":"de8d947b57d76c3c21840c240c523e7fd950b18de17efe3c50b6dd45b3f8ea26"}
Nov 02 13:21:26.851475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:26.851414    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq" podStartSLOduration=1.8513986340000002 podStartE2EDuration="1.851398634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:21:26.850618701 +0000 UTC m=+3232.957321552" watchObservedRunningTime="2026-11-02 13:21:26.851398634 +0000 UTC m=+3232.958101485"
Nov 02 13:21:27.147560 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:27.147524    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:27.254610 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:27.254563    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:28.132492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:28.132457    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:29.854106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:29.854071    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:31.309647 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:31.309601    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:32.056261 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:32.056225    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:33.161609 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.161571    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:33.289951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.289913    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:33.367813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.367742    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2" podUID="9b74e33e-5277-41e1-a362-210e3d888b2e" containerName="main" containerID="cri-o://47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975" gracePeriod=45
Nov 02 13:21:33.678411 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.678386    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:21:33.707767 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.707739    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-dshm\") pod \"9b74e33e-5277-41e1-a362-210e3d888b2e\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") "
Nov 02 13:21:33.707965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.707871    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b74e33e-5277-41e1-a362-210e3d888b2e-kube-api-access-sb86j\" (UniqueName: \"kubernetes.io/projected/9b74e33e-5277-41e1-a362-210e3d888b2e-kube-api-access-sb86j\") pod \"9b74e33e-5277-41e1-a362-210e3d888b2e\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") "
Nov 02 13:21:33.707965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.707924    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-model-cache\") pod \"9b74e33e-5277-41e1-a362-210e3d888b2e\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") "
Nov 02 13:21:33.708124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.707986    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-home\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-home\") pod \"9b74e33e-5277-41e1-a362-210e3d888b2e\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") "
Nov 02 13:21:33.708124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.708090    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b74e33e-5277-41e1-a362-210e3d888b2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b74e33e-5277-41e1-a362-210e3d888b2e-tls-certs\") pod \"9b74e33e-5277-41e1-a362-210e3d888b2e\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") "
Nov 02 13:21:33.708243 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.708126    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-tmp-dir\") pod \"9b74e33e-5277-41e1-a362-210e3d888b2e\" (UID: \"9b74e33e-5277-41e1-a362-210e3d888b2e\") "
Nov 02 13:21:33.708305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.708281    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-model-cache" pod "9b74e33e-5277-41e1-a362-210e3d888b2e" (UID: "9b74e33e-5277-41e1-a362-210e3d888b2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:33.708401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.708373    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-home" pod "9b74e33e-5277-41e1-a362-210e3d888b2e" (UID: "9b74e33e-5277-41e1-a362-210e3d888b2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:33.708579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.708557    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-tmp-dir" pod "9b74e33e-5277-41e1-a362-210e3d888b2e" (UID: "9b74e33e-5277-41e1-a362-210e3d888b2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:33.709220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.709193    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:33.709341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.709223    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:33.709341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.709238    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:33.710581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.710551    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-dshm" pod "9b74e33e-5277-41e1-a362-210e3d888b2e" (UID: "9b74e33e-5277-41e1-a362-210e3d888b2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:33.710717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.710691    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b74e33e-5277-41e1-a362-210e3d888b2e-kube-api-access-sb86j" pod "9b74e33e-5277-41e1-a362-210e3d888b2e" (UID: "9b74e33e-5277-41e1-a362-210e3d888b2e"). InnerVolumeSpecName "kube-api-access-sb86j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:33.710857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.710835    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b74e33e-5277-41e1-a362-210e3d888b2e-tls-certs" pod "9b74e33e-5277-41e1-a362-210e3d888b2e" (UID: "9b74e33e-5277-41e1-a362-210e3d888b2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:21:33.776959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.776866    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:33.810759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.810708    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b74e33e-5277-41e1-a362-210e3d888b2e-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:33.810759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.810754    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb86j\" (UniqueName: \"kubernetes.io/projected/9b74e33e-5277-41e1-a362-210e3d888b2e-kube-api-access-sb86j\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:33.811040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.810771    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b74e33e-5277-41e1-a362-210e3d888b2e-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:33.878708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.878675    5879 generic.go:356] "Generic (PLEG): container finished" podID="9b74e33e-5277-41e1-a362-210e3d888b2e" containerID="47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975" exitCode=0
Nov 02 13:21:33.878899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.878723    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"
Nov 02 13:21:33.878899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.878765    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2" event={"ID":"9b74e33e-5277-41e1-a362-210e3d888b2e","Type":"ContainerDied","Data":"47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975"}
Nov 02 13:21:33.878899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.878815    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2" event={"ID":"9b74e33e-5277-41e1-a362-210e3d888b2e","Type":"ContainerDied","Data":"f7c3d703afb90de6e5e92eb7be07230b7631db3354442d9ef5ffa2e893e461a1"}
Nov 02 13:21:33.878899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.878834    5879 scope.go:122] "RemoveContainer" containerID="47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975"
Nov 02 13:21:33.894555 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.894523    5879 scope.go:122] "RemoveContainer" containerID="47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975"
Nov 02 13:21:33.894889 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:21:33.894864    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975\": container with ID starting with 47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975 not found: ID does not exist" containerID="47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975"
Nov 02 13:21:33.895009 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.894899    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975"} err="failed to get container status \"47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975\": rpc error: code = NotFound desc = could not find container \"47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975\": container with ID starting with 47c5dde0ea615fef5701a3accce3385b74c826f1e9b659cce38ab02fe3825975 not found: ID does not exist"
Nov 02 13:21:33.967578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.967543    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"]
Nov 02 13:21:33.979252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.976916    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:33.980216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:33.980186    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6563ed/test-model-6563ed-kserve-546b9748d5-xgqz2"]
Nov 02 13:21:34.027343 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:34.027243    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b74e33e-5277-41e1-a362-210e3d888b2e" path="/var/lib/kubelet/pods/9b74e33e-5277-41e1-a362-210e3d888b2e/volumes"
Nov 02 13:21:34.336150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:34.336041    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"]
Nov 02 13:21:34.336952 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:34.336914    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl" podUID="f7768beb-47c2-4a5d-a281-75e09a47965b" containerName="istio-proxy" containerID="cri-o://8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b" gracePeriod=30
Nov 02 13:21:34.391282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:34.391248    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:34.951680 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:34.951651    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:35.429063 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:35.429029    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:35.585826 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:35.585788    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:35.745066 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:35.744975    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:36.254561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:36.254528    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:36.267906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:36.267876    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:36.922683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:36.922652    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:21:37.343407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:37.343311    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:39.007233 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.007197    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:39.036837 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.036800    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"]
Nov 02 13:21:39.048879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.048840    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9817a0b2-d4p4c"]
Nov 02 13:21:39.638105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.638077    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:21:39.721303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.721265    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-data\") pod \"f7768beb-47c2-4a5d-a281-75e09a47965b\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") "
Nov 02 13:21:39.721576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.721551    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-certs\") pod \"f7768beb-47c2-4a5d-a281-75e09a47965b\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") "
Nov 02 13:21:39.721730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.721599    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-credential-socket\") pod \"f7768beb-47c2-4a5d-a281-75e09a47965b\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") "
Nov 02 13:21:39.721730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.721626    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-envoy\") pod \"f7768beb-47c2-4a5d-a281-75e09a47965b\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") "
Nov 02 13:21:39.721730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.721657    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7768beb-47c2-4a5d-a281-75e09a47965b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7768beb-47c2-4a5d-a281-75e09a47965b-istiod-ca-cert\") pod \"f7768beb-47c2-4a5d-a281-75e09a47965b\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") "
Nov 02 13:21:39.721730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.721685    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-kube-api-access-92j94\" (UniqueName: \"kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-kube-api-access-92j94\") pod \"f7768beb-47c2-4a5d-a281-75e09a47965b\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") "
Nov 02 13:21:39.722525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.721778    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-podinfo\") pod \"f7768beb-47c2-4a5d-a281-75e09a47965b\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") "
Nov 02 13:21:39.722525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.721838    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-socket\") pod \"f7768beb-47c2-4a5d-a281-75e09a47965b\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") "
Nov 02 13:21:39.722525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.721885    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-token\" (UniqueName: \"kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-token\") pod \"f7768beb-47c2-4a5d-a281-75e09a47965b\" (UID: \"f7768beb-47c2-4a5d-a281-75e09a47965b\") "
Nov 02 13:21:39.722525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.722435    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7768beb-47c2-4a5d-a281-75e09a47965b-istiod-ca-cert" pod "f7768beb-47c2-4a5d-a281-75e09a47965b" (UID: "f7768beb-47c2-4a5d-a281-75e09a47965b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:21:39.722785 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.722737    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7768beb-47c2-4a5d-a281-75e09a47965b-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.723153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.723129    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-certs" pod "f7768beb-47c2-4a5d-a281-75e09a47965b" (UID: "f7768beb-47c2-4a5d-a281-75e09a47965b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:39.723311 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.723292    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-socket" pod "f7768beb-47c2-4a5d-a281-75e09a47965b" (UID: "f7768beb-47c2-4a5d-a281-75e09a47965b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:39.724287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.724032    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-credential-socket" pod "f7768beb-47c2-4a5d-a281-75e09a47965b" (UID: "f7768beb-47c2-4a5d-a281-75e09a47965b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:39.724613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.724595    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-data" pod "f7768beb-47c2-4a5d-a281-75e09a47965b" (UID: "f7768beb-47c2-4a5d-a281-75e09a47965b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:39.726047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.725965    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-podinfo" pod "f7768beb-47c2-4a5d-a281-75e09a47965b" (UID: "f7768beb-47c2-4a5d-a281-75e09a47965b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:21:39.726295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.726268    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-token" pod "f7768beb-47c2-4a5d-a281-75e09a47965b" (UID: "f7768beb-47c2-4a5d-a281-75e09a47965b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:39.727246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.727216    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-kube-api-access-92j94" pod "f7768beb-47c2-4a5d-a281-75e09a47965b" (UID: "f7768beb-47c2-4a5d-a281-75e09a47965b"). InnerVolumeSpecName "kube-api-access-92j94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:39.727691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.727668    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-envoy" pod "f7768beb-47c2-4a5d-a281-75e09a47965b" (UID: "f7768beb-47c2-4a5d-a281-75e09a47965b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:21:39.824799 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.824760    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.824799 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.824797    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.825103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.824810    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.825103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.824824    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92j94\" (UniqueName: \"kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-kube-api-access-92j94\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.825103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.824835    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.825103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.824847    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.825103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.824858    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.825103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.824868    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7768beb-47c2-4a5d-a281-75e09a47965b-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:39.941012 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.940961    5879 generic.go:356] "Generic (PLEG): container finished" podID="f7768beb-47c2-4a5d-a281-75e09a47965b" containerID="8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b" exitCode=0
Nov 02 13:21:39.941224 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.941044    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"
Nov 02 13:21:39.941224 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.941053    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl" event={"ID":"f7768beb-47c2-4a5d-a281-75e09a47965b","Type":"ContainerDied","Data":"8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b"}
Nov 02 13:21:39.941224 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.941119    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl" event={"ID":"f7768beb-47c2-4a5d-a281-75e09a47965b","Type":"ContainerDied","Data":"bf894c355dfb6bc3937c1038a7aae2de68094cc1aeb067dbd5cee690ca31b9d0"}
Nov 02 13:21:39.941224 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.941132    5879 scope.go:122] "RemoveContainer" containerID="8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b"
Nov 02 13:21:39.953509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.953490    5879 scope.go:122] "RemoveContainer" containerID="8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b"
Nov 02 13:21:39.953808 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:21:39.953788    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b\": container with ID starting with 8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b not found: ID does not exist" containerID="8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b"
Nov 02 13:21:39.953861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:39.953816    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b"} err="failed to get container status \"8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b\": rpc error: code = NotFound desc = could not find container \"8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b\": container with ID starting with 8a39e2214b8a5028df9c93792bb8d777d9302e4247f9243da195daf90ff0df0b not found: ID does not exist"
Nov 02 13:21:40.003936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:40.003899    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"]
Nov 02 13:21:40.009494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:40.009458    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9139ec-openshift-default-58b88c89b5-525zl"]
Nov 02 13:21:40.021054 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:40.021020    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac0e3844-a606-4544-a56d-e243353f397e" path="/var/lib/kubelet/pods/ac0e3844-a606-4544-a56d-e243353f397e/volumes"
Nov 02 13:21:40.021601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:40.021586    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7768beb-47c2-4a5d-a281-75e09a47965b" path="/var/lib/kubelet/pods/f7768beb-47c2-4a5d-a281-75e09a47965b/volumes"
Nov 02 13:21:40.195813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:40.195725    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:40.315686 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:40.315648    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:40.669175 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:40.669142    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:40.910495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:40.910466    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:41.364136 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.364102    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"]
Nov 02 13:21:41.366875 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.366842    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9b74e33e-5277-41e1-a362-210e3d888b2e" containerName="main"
Nov 02 13:21:41.367101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.367083    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b74e33e-5277-41e1-a362-210e3d888b2e" containerName="main"
Nov 02 13:21:41.367284 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.367271    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f7768beb-47c2-4a5d-a281-75e09a47965b" containerName="istio-proxy"
Nov 02 13:21:41.367400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.367388    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7768beb-47c2-4a5d-a281-75e09a47965b" containerName="istio-proxy"
Nov 02 13:21:41.368075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.368055    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b74e33e-5277-41e1-a362-210e3d888b2e" containerName="main"
Nov 02 13:21:41.368219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.368208    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7768beb-47c2-4a5d-a281-75e09a47965b" containerName="istio-proxy"
Nov 02 13:21:41.373414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.373395    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:41.390594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.390566    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"]
Nov 02 13:21:41.397779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.397749    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:21:41.397893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.397810    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:21:41.489508 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.489475    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-957c3d45-mdbg6\" (UID: \"52871ebe-9271-4d59-a2ac-e6c372c88eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:41.489714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.489660    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr7k9\" (UniqueName: \"kubernetes.io/projected/52871ebe-9271-4d59-a2ac-e6c372c88eae-kube-api-access-qr7k9\") pod \"maas-api-revoke-sub-957c3d45-mdbg6\" (UID: \"52871ebe-9271-4d59-a2ac-e6c372c88eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:41.489714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.489709    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-957c3d45-mdbg6\" (UID: \"52871ebe-9271-4d59-a2ac-e6c372c88eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:41.590960 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.590919    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-957c3d45-mdbg6\" (UID: \"52871ebe-9271-4d59-a2ac-e6c372c88eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:41.591185 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.591118    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7k9\" (UniqueName: \"kubernetes.io/projected/52871ebe-9271-4d59-a2ac-e6c372c88eae-kube-api-access-qr7k9\") pod \"maas-api-revoke-sub-957c3d45-mdbg6\" (UID: \"52871ebe-9271-4d59-a2ac-e6c372c88eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:41.591185 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.591167    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-957c3d45-mdbg6\" (UID: \"52871ebe-9271-4d59-a2ac-e6c372c88eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:41.591779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.591750    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-957c3d45-mdbg6\" (UID: \"52871ebe-9271-4d59-a2ac-e6c372c88eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:41.593984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.593956    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-957c3d45-mdbg6\" (UID: \"52871ebe-9271-4d59-a2ac-e6c372c88eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:41.616706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.616637    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7k9\" (UniqueName: \"kubernetes.io/projected/52871ebe-9271-4d59-a2ac-e6c372c88eae-kube-api-access-qr7k9\") pod \"maas-api-revoke-sub-957c3d45-mdbg6\" (UID: \"52871ebe-9271-4d59-a2ac-e6c372c88eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:41.694947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.694911    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:41.887267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:41.887179    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:42.043073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:42.043047    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"]
Nov 02 13:21:42.046563 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:21:42.046537    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52871ebe_9271_4d59_a2ac_e6c372c88eae.slice/crio-bcce08a75df08403aab12ebe346151e93ddf166dfd02dcf62c9bef568fad168f WatchSource:0}: Error finding container bcce08a75df08403aab12ebe346151e93ddf166dfd02dcf62c9bef568fad168f: Status 404 returned error can't find the container with id bcce08a75df08403aab12ebe346151e93ddf166dfd02dcf62c9bef568fad168f
Nov 02 13:21:42.358199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:42.358150    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:42.567301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:42.567214    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:43.029414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:43.029379    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6" event={"ID":"52871ebe-9271-4d59-a2ac-e6c372c88eae","Type":"ContainerStarted","Data":"8df4d9430d7948938fb57ddcb47a2ada5d38492d941266e25b536a4906ef6e18"}
Nov 02 13:21:43.029595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:43.029429    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6" event={"ID":"52871ebe-9271-4d59-a2ac-e6c372c88eae","Type":"ContainerStarted","Data":"bcce08a75df08403aab12ebe346151e93ddf166dfd02dcf62c9bef568fad168f"}
Nov 02 13:21:43.080000 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:43.079941    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6" podStartSLOduration=2.079926913 podStartE2EDuration="2.079926913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:21:43.075605399 +0000 UTC m=+3249.182308275" watchObservedRunningTime="2026-11-02 13:21:43.079926913 +0000 UTC m=+3249.186629764"
Nov 02 13:21:43.985503 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:43.985420    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:44.109013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:44.097650    5879 generic.go:356] "Generic (PLEG): container finished" podID="52871ebe-9271-4d59-a2ac-e6c372c88eae" containerID="8df4d9430d7948938fb57ddcb47a2ada5d38492d941266e25b536a4906ef6e18" exitCode=0
Nov 02 13:21:44.109013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:44.097838    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6" event={"ID":"52871ebe-9271-4d59-a2ac-e6c372c88eae","Type":"ContainerDied","Data":"8df4d9430d7948938fb57ddcb47a2ada5d38492d941266e25b536a4906ef6e18"}
Nov 02 13:21:44.826786 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:21:44.826746    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:21:45.298139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:45.298109    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:45.370735 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:45.370699    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52871ebe-9271-4d59-a2ac-e6c372c88eae-kube-api-access-qr7k9\" (UniqueName: \"kubernetes.io/projected/52871ebe-9271-4d59-a2ac-e6c372c88eae-kube-api-access-qr7k9\") pod \"52871ebe-9271-4d59-a2ac-e6c372c88eae\" (UID: \"52871ebe-9271-4d59-a2ac-e6c372c88eae\") "
Nov 02 13:21:45.370930 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:45.370911    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-openshift-ca\") pod \"52871ebe-9271-4d59-a2ac-e6c372c88eae\" (UID: \"52871ebe-9271-4d59-a2ac-e6c372c88eae\") "
Nov 02 13:21:45.371036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:45.370958    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-xks-ca\") pod \"52871ebe-9271-4d59-a2ac-e6c372c88eae\" (UID: \"52871ebe-9271-4d59-a2ac-e6c372c88eae\") "
Nov 02 13:21:45.371318 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:45.371292    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-openshift-ca" pod "52871ebe-9271-4d59-a2ac-e6c372c88eae" (UID: "52871ebe-9271-4d59-a2ac-e6c372c88eae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:21:45.371820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:45.371799    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:45.373550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:45.373524    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52871ebe-9271-4d59-a2ac-e6c372c88eae-kube-api-access-qr7k9" pod "52871ebe-9271-4d59-a2ac-e6c372c88eae" (UID: "52871ebe-9271-4d59-a2ac-e6c372c88eae"). InnerVolumeSpecName "kube-api-access-qr7k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:21:45.373864 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:45.373838    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-xks-ca" pod "52871ebe-9271-4d59-a2ac-e6c372c88eae" (UID: "52871ebe-9271-4d59-a2ac-e6c372c88eae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:21:45.473419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:45.473372    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52871ebe-9271-4d59-a2ac-e6c372c88eae-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:45.473419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:45.473408    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr7k9\" (UniqueName: \"kubernetes.io/projected/52871ebe-9271-4d59-a2ac-e6c372c88eae-kube-api-access-qr7k9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:21:45.779196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:45.779108    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:46.116984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:46.116950    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"
Nov 02 13:21:46.116984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:46.117005    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6" event={"ID":"52871ebe-9271-4d59-a2ac-e6c372c88eae","Type":"ContainerDied","Data":"bcce08a75df08403aab12ebe346151e93ddf166dfd02dcf62c9bef568fad168f"}
Nov 02 13:21:46.117251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:46.117064    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcce08a75df08403aab12ebe346151e93ddf166dfd02dcf62c9bef568fad168f"
Nov 02 13:21:47.747613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:47.747553    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:48.556056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.556016    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"]
Nov 02 13:21:48.557423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.557395    5879 cpu_manager.go:414] "containerMap: removing container" podUID="52871ebe-9271-4d59-a2ac-e6c372c88eae" containerName="revoke-keys"
Nov 02 13:21:48.557423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.557419    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52871ebe-9271-4d59-a2ac-e6c372c88eae" containerName="revoke-keys"
Nov 02 13:21:48.557734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.557716    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="52871ebe-9271-4d59-a2ac-e6c372c88eae" containerName="revoke-keys"
Nov 02 13:21:48.561612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.561591    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.578769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.578587    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-23f720\"/\"openshift-service-ca.crt\""
Nov 02 13:21:48.578896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.578786    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-23f720\"/\"kube-root-ca.crt\""
Nov 02 13:21:48.578896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.578880    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-23f720\"/\"test-model-23f720-kserve-self-signed-certs\""
Nov 02 13:21:48.579093 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.579071    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-23f720\"/\"default-dockercfg-lqkvm\""
Nov 02 13:21:48.585885 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.585854    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"]
Nov 02 13:21:48.623740 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.623683    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-home\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.623964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.623787    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tls-certs\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.623964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.623914    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-dshm\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.624134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.623982    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-model-cache\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.624199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.624131    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk5rg\" (UniqueName: \"kubernetes.io/projected/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-kube-api-access-nk5rg\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.624199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.624162    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tmp-dir\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.629091 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.629065    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:48.726100 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.726044    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tls-certs\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.726306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.726202    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-dshm\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.726306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.726300    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-model-cache\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.726478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.726438    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk5rg\" (UniqueName: \"kubernetes.io/projected/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-kube-api-access-nk5rg\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.726478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.726472    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tmp-dir\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.727040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.726656    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-home\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.727040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.726710    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-model-cache\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.727040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.726950    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tmp-dir\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.727302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.727065    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-home\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.728951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.728928    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tls-certs\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.729255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.729232    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-dshm\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.755849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.755815    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk5rg\" (UniqueName: \"kubernetes.io/projected/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-kube-api-access-nk5rg\") pod \"test-model-23f720-kserve-7f4bbdcf75-5jpl9\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") " pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:48.883746 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:48.883700    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:49.213725 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:21:49.213686    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4debb049_ebb2_4e5b_932a_04bfc8dd91c8.slice/crio-7b43b2bc35979b601b8ab8c69b5c8e0b4a70223bfcfdb273dfe33f7c117948e2 WatchSource:0}: Error finding container 7b43b2bc35979b601b8ab8c69b5c8e0b4a70223bfcfdb273dfe33f7c117948e2: Status 404 returned error can't find the container with id 7b43b2bc35979b601b8ab8c69b5c8e0b4a70223bfcfdb273dfe33f7c117948e2
Nov 02 13:21:49.216611 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:49.216583    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"]
Nov 02 13:21:49.563692 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:49.563612    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:49.693943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:49.693918    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:50.035480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:50.035435    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"]
Nov 02 13:21:50.080018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:50.057199    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52e0d30e-pmz82"]
Nov 02 13:21:50.195356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:50.190653    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9" event={"ID":"4debb049-ebb2-4e5b-932a-04bfc8dd91c8","Type":"ContainerStarted","Data":"24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e"}
Nov 02 13:21:50.195356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:50.190726    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9" event={"ID":"4debb049-ebb2-4e5b-932a-04bfc8dd91c8","Type":"ContainerStarted","Data":"7b43b2bc35979b601b8ab8c69b5c8e0b4a70223bfcfdb273dfe33f7c117948e2"}
Nov 02 13:21:50.240679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:50.240620    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9" podStartSLOduration=2.240604949 podStartE2EDuration="2.240604949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:21:50.239452246 +0000 UTC m=+3256.346155101" watchObservedRunningTime="2026-11-02 13:21:50.240604949 +0000 UTC m=+3256.347307800"
Nov 02 13:21:50.884419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:50.884384    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:51.232819 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:51.232712    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:52.020581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:52.020545    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37121a77-285c-4f67-81fa-00b4ba4b2288" path="/var/lib/kubelet/pods/37121a77-285c-4f67-81fa-00b4ba4b2288/volumes"
Nov 02 13:21:52.836201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:52.836165    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:53.554695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:53.554638    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:53.937044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:53.937008    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"]
Nov 02 13:21:53.995804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:53.995765    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:54.465833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:54.465801    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:54.667120 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:54.667084    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:55.823362 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:55.823319    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:56.084462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:56.084369    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:57.768657 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:57.768623    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:21:58.885098 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:58.885061    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:58.898495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:58.898473    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:59.279728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:59.279648    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:21:59.539338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:21:59.539228    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:00.111986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:00.111953    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"]
Nov 02 13:22:00.295921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:00.295843    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:00.424110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:00.424075    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:01.338334 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:01.338300    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:01.471859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:01.471824    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:02.286277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.286247    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"]
Nov 02 13:22:02.310891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.310863    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.316591 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.316557    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"]
Nov 02 13:22:02.341910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.341885    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8962e9\"/\"default-dockercfg-5dptl\""
Nov 02 13:22:02.342339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.342308    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8962e9\"/\"openshift-service-ca.crt\""
Nov 02 13:22:02.342418 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.342354    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8962e9\"/\"test-model-8962e9-kserve-self-signed-certs\""
Nov 02 13:22:02.342418 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.342361    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8962e9\"/\"kube-root-ca.crt\""
Nov 02 13:22:02.367426 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.367387    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:02.404069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.403975    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9327c2b-5304-4513-b21c-0f9174e60ff0-tls-certs\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.404278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.404172    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-model-cache\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.404278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.404208    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-dshm\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.404278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.404234    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjfkd\" (UniqueName: \"kubernetes.io/projected/c9327c2b-5304-4513-b21c-0f9174e60ff0-kube-api-access-xjfkd\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.404482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.404450    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-tmp-dir\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.404544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.404509    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-home\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.506397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.506361    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-tmp-dir\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.506605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.506448    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-home\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.506691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.506605    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9327c2b-5304-4513-b21c-0f9174e60ff0-tls-certs\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.506763 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.506720    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-model-cache\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.506763 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.506754    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-dshm\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.506877 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.506783    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjfkd\" (UniqueName: \"kubernetes.io/projected/c9327c2b-5304-4513-b21c-0f9174e60ff0-kube-api-access-xjfkd\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.506877 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.506816    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-tmp-dir\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.507270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.507245    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-model-cache\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.507562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.507541    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-home\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.510500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.510476    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9327c2b-5304-4513-b21c-0f9174e60ff0-tls-certs\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.511551 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.511527    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-dshm\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.536958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.536842    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjfkd\" (UniqueName: \"kubernetes.io/projected/c9327c2b-5304-4513-b21c-0f9174e60ff0-kube-api-access-xjfkd\") pod \"test-model-8962e9-kserve-64674c5bc7-5mpbw\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") " pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.631593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.631304    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:02.956650 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.956620    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:02.959290 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:22:02.959252    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9327c2b_5304_4513_b21c_0f9174e60ff0.slice/crio-5f8be04e742a3303b0e611df9abb24acfa4330c1dbd572b127abc83f3fc73e37 WatchSource:0}: Error finding container 5f8be04e742a3303b0e611df9abb24acfa4330c1dbd572b127abc83f3fc73e37: Status 404 returned error can't find the container with id 5f8be04e742a3303b0e611df9abb24acfa4330c1dbd572b127abc83f3fc73e37
Nov 02 13:22:02.960471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:02.960444    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"]
Nov 02 13:22:03.334503 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:03.334394    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw" event={"ID":"c9327c2b-5304-4513-b21c-0f9174e60ff0","Type":"ContainerStarted","Data":"850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c"}
Nov 02 13:22:03.334503 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:03.334450    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw" event={"ID":"c9327c2b-5304-4513-b21c-0f9174e60ff0","Type":"ContainerStarted","Data":"5f8be04e742a3303b0e611df9abb24acfa4330c1dbd572b127abc83f3fc73e37"}
Nov 02 13:22:03.384545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:03.384472    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw" podStartSLOduration=1.3844369109999999 podStartE2EDuration="1.384436911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:22:03.382382538 +0000 UTC m=+3269.489085389" watchObservedRunningTime="2026-11-02 13:22:03.384436911 +0000 UTC m=+3269.491139763"
Nov 02 13:22:03.631931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:03.631895    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:03.654507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:03.654475    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:05.112406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:05.112358    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68" podUID="f00ea282-bd5f-4e08-8def-89fb409a4d24" containerName="payload-processing" containerID="cri-o://fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe" gracePeriod=25
Nov 02 13:22:05.361696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:05.361663    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:06.032959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:06.032919    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"]
Nov 02 13:22:06.047013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:06.046958    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f99a4a0d-hskpk"]
Nov 02 13:22:06.890372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:06.890341    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:07.566842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:07.566801    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68" podUID="f00ea282-bd5f-4e08-8def-89fb409a4d24" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:22:07.837044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:07.836930    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:08.020233 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:08.020195    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04769618-7fd0-4a6d-8598-93e5a118abbc" path="/var/lib/kubelet/pods/04769618-7fd0-4a6d-8598-93e5a118abbc/volumes"
Nov 02 13:22:08.029432 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:08.029397    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:08.558142 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:08.558102    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:08.685240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:08.685205    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:08.964414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:08.964378    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"]
Nov 02 13:22:08.964666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:08.964634    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm" podUID="c77bd7e8-4b52-427b-90c5-ad7460330a8a" containerName="istio-proxy" containerID="cri-o://77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369" gracePeriod=30
Nov 02 13:22:08.976074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:08.975972    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2" podUID="e168fd9f-aa07-4650-b616-37ee7b92ee77" containerName="main" containerID="cri-o://84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58" gracePeriod=45
Nov 02 13:22:09.011065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.009142    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:09.369733 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.369701    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:22:09.388983 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.388946    5879 generic.go:356] "Generic (PLEG): container finished" podID="e168fd9f-aa07-4650-b616-37ee7b92ee77" containerID="84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58" exitCode=0
Nov 02 13:22:09.389223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.389038    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"
Nov 02 13:22:09.389223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.389047    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2" event={"ID":"e168fd9f-aa07-4650-b616-37ee7b92ee77","Type":"ContainerDied","Data":"84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58"}
Nov 02 13:22:09.389223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.389095    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2" event={"ID":"e168fd9f-aa07-4650-b616-37ee7b92ee77","Type":"ContainerDied","Data":"43fb440c066afeec1190d55c6d50a952a01d43139b3db13bf335239fadc33f7b"}
Nov 02 13:22:09.389223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.389129    5879 scope.go:122] "RemoveContainer" containerID="84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58"
Nov 02 13:22:09.404920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.404901    5879 scope.go:122] "RemoveContainer" containerID="84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58"
Nov 02 13:22:09.405297 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:22:09.405278    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58\": container with ID starting with 84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58 not found: ID does not exist" containerID="84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58"
Nov 02 13:22:09.405387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.405308    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58"} err="failed to get container status \"84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58\": rpc error: code = NotFound desc = could not find container \"84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58\": container with ID starting with 84a62503bafd82d9f7eb9c8845d5afd9a54e54a880ebfc25c6ceda304e57ca58 not found: ID does not exist"
Nov 02 13:22:09.452044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.451981    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-home\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-home\") pod \"e168fd9f-aa07-4650-b616-37ee7b92ee77\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") "
Nov 02 13:22:09.452252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.452106    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-model-cache\") pod \"e168fd9f-aa07-4650-b616-37ee7b92ee77\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") "
Nov 02 13:22:09.452334 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.452268    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e168fd9f-aa07-4650-b616-37ee7b92ee77-tls-certs\" (UniqueName: \"kubernetes.io/secret/e168fd9f-aa07-4650-b616-37ee7b92ee77-tls-certs\") pod \"e168fd9f-aa07-4650-b616-37ee7b92ee77\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") "
Nov 02 13:22:09.452403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.452327    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-home" pod "e168fd9f-aa07-4650-b616-37ee7b92ee77" (UID: "e168fd9f-aa07-4650-b616-37ee7b92ee77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:09.452403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.452337    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e168fd9f-aa07-4650-b616-37ee7b92ee77-kube-api-access-kzmjk\" (UniqueName: \"kubernetes.io/projected/e168fd9f-aa07-4650-b616-37ee7b92ee77-kube-api-access-kzmjk\") pod \"e168fd9f-aa07-4650-b616-37ee7b92ee77\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") "
Nov 02 13:22:09.452531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.452483    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-tmp-dir\") pod \"e168fd9f-aa07-4650-b616-37ee7b92ee77\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") "
Nov 02 13:22:09.452628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.452607    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-dshm\") pod \"e168fd9f-aa07-4650-b616-37ee7b92ee77\" (UID: \"e168fd9f-aa07-4650-b616-37ee7b92ee77\") "
Nov 02 13:22:09.453024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.452851    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-tmp-dir" pod "e168fd9f-aa07-4650-b616-37ee7b92ee77" (UID: "e168fd9f-aa07-4650-b616-37ee7b92ee77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:09.453369 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.453340    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-model-cache" pod "e168fd9f-aa07-4650-b616-37ee7b92ee77" (UID: "e168fd9f-aa07-4650-b616-37ee7b92ee77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:09.453582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.453559    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:09.453628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.453589    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:09.453628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.453604    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:09.455518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.455486    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e168fd9f-aa07-4650-b616-37ee7b92ee77-tls-certs" pod "e168fd9f-aa07-4650-b616-37ee7b92ee77" (UID: "e168fd9f-aa07-4650-b616-37ee7b92ee77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:22:09.455518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.455492    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e168fd9f-aa07-4650-b616-37ee7b92ee77-kube-api-access-kzmjk" pod "e168fd9f-aa07-4650-b616-37ee7b92ee77" (UID: "e168fd9f-aa07-4650-b616-37ee7b92ee77"). InnerVolumeSpecName "kube-api-access-kzmjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:09.455931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.455911    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-dshm" pod "e168fd9f-aa07-4650-b616-37ee7b92ee77" (UID: "e168fd9f-aa07-4650-b616-37ee7b92ee77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:09.554739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.554653    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e168fd9f-aa07-4650-b616-37ee7b92ee77-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:09.554739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.554683    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzmjk\" (UniqueName: \"kubernetes.io/projected/e168fd9f-aa07-4650-b616-37ee7b92ee77-kube-api-access-kzmjk\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:09.554739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.554692    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e168fd9f-aa07-4650-b616-37ee7b92ee77-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:09.763032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.762983    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"]
Nov 02 13:22:09.794982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.794946    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-48d74e/test-model-48d74e-kserve-85dcfc4f5f-rtsl2"]
Nov 02 13:22:09.865113 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:09.865076    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:10.019545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:10.019509    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e168fd9f-aa07-4650-b616-37ee7b92ee77" path="/var/lib/kubelet/pods/e168fd9f-aa07-4650-b616-37ee7b92ee77/volumes"
Nov 02 13:22:10.486214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:10.486166    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:10.586124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:10.586082    5879 patch_prober.go:29] interesting pod/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:22:10.586336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:10.586142    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm" podUID="c77bd7e8-4b52-427b-90c5-ad7460330a8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:22:12.110793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:12.110756    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:12.632156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:12.632123    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:12.646025 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:12.645963    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:13.436451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:13.436415    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:22:13.740442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:13.740355    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:14.264256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.264188    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:22:14.335136 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.335100    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-podinfo\") pod \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") "
Nov 02 13:22:14.335341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.335232    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-socket\") pod \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") "
Nov 02 13:22:14.335341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.335299    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-kube-api-access-vh27b\" (UniqueName: \"kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-kube-api-access-vh27b\") pod \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") "
Nov 02 13:22:14.335341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.335332    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-certs\") pod \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") "
Nov 02 13:22:14.335531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.335382    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-token\" (UniqueName: \"kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-token\") pod \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") "
Nov 02 13:22:14.335531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.335458    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-envoy\") pod \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") "
Nov 02 13:22:14.335640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.335607    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-data\") pod \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") "
Nov 02 13:22:14.336961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.335675    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istiod-ca-cert\") pod \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") "
Nov 02 13:22:14.336961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.335725    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-credential-socket\") pod \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\" (UID: \"c77bd7e8-4b52-427b-90c5-ad7460330a8a\") "
Nov 02 13:22:14.336961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.336444    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istiod-ca-cert" pod "c77bd7e8-4b52-427b-90c5-ad7460330a8a" (UID: "c77bd7e8-4b52-427b-90c5-ad7460330a8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:14.336961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.336899    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.342852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.338912    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-certs" pod "c77bd7e8-4b52-427b-90c5-ad7460330a8a" (UID: "c77bd7e8-4b52-427b-90c5-ad7460330a8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.342852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.338979    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-data" pod "c77bd7e8-4b52-427b-90c5-ad7460330a8a" (UID: "c77bd7e8-4b52-427b-90c5-ad7460330a8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.342852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.339511    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-credential-socket" pod "c77bd7e8-4b52-427b-90c5-ad7460330a8a" (UID: "c77bd7e8-4b52-427b-90c5-ad7460330a8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.342852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.339577    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-socket" pod "c77bd7e8-4b52-427b-90c5-ad7460330a8a" (UID: "c77bd7e8-4b52-427b-90c5-ad7460330a8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.342852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.341764    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-kube-api-access-vh27b" pod "c77bd7e8-4b52-427b-90c5-ad7460330a8a" (UID: "c77bd7e8-4b52-427b-90c5-ad7460330a8a"). InnerVolumeSpecName "kube-api-access-vh27b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:14.345572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.345531    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-envoy" pod "c77bd7e8-4b52-427b-90c5-ad7460330a8a" (UID: "c77bd7e8-4b52-427b-90c5-ad7460330a8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:14.345872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.345739    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-token" pod "c77bd7e8-4b52-427b-90c5-ad7460330a8a" (UID: "c77bd7e8-4b52-427b-90c5-ad7460330a8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:14.347286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.347249    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-podinfo" pod "c77bd7e8-4b52-427b-90c5-ad7460330a8a" (UID: "c77bd7e8-4b52-427b-90c5-ad7460330a8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:14.437695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.437646    5879 generic.go:356] "Generic (PLEG): container finished" podID="c77bd7e8-4b52-427b-90c5-ad7460330a8a" containerID="77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369" exitCode=0
Nov 02 13:22:14.438281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.437779    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"
Nov 02 13:22:14.438281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.437799    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm" event={"ID":"c77bd7e8-4b52-427b-90c5-ad7460330a8a","Type":"ContainerDied","Data":"77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369"}
Nov 02 13:22:14.438281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.437872    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm" event={"ID":"c77bd7e8-4b52-427b-90c5-ad7460330a8a","Type":"ContainerDied","Data":"3da82186d464367370c8382298e80f7ff8e58c5db6ffa358a859470c503229ee"}
Nov 02 13:22:14.438281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.437904    5879 scope.go:122] "RemoveContainer" containerID="77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369"
Nov 02 13:22:14.438505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.438422    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.438646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.438622    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh27b\" (UniqueName: \"kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-kube-api-access-vh27b\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.438775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.438653    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.438775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.438669    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.438775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.438684    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.438775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.438700    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.438775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.438716    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c77bd7e8-4b52-427b-90c5-ad7460330a8a-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.439107 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.438843    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c77bd7e8-4b52-427b-90c5-ad7460330a8a-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:14.452962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.452940    5879 scope.go:122] "RemoveContainer" containerID="77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369"
Nov 02 13:22:14.453381 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:22:14.453343    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369\": container with ID starting with 77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369 not found: ID does not exist" containerID="77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369"
Nov 02 13:22:14.453564 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.453386    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369"} err="failed to get container status \"77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369\": rpc error: code = NotFound desc = could not find container \"77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369\": container with ID starting with 77cc9404c74103adea38ceb5fff04a0fd866a3b67a45a9c42523e1c24054e369 not found: ID does not exist"
Nov 02 13:22:14.505389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.505340    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"]
Nov 02 13:22:14.509356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:14.509321    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5f8899-openshift-default-67cc5d8789-xprnm"]
Nov 02 13:22:14.823206 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:22:14.823170    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:22:15.361169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:15.361128    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:16.019624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:16.019594    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c77bd7e8-4b52-427b-90c5-ad7460330a8a" path="/var/lib/kubelet/pods/c77bd7e8-4b52-427b-90c5-ad7460330a8a/volumes"
Nov 02 13:22:16.224174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:16.224138    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:16.344462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:16.344364    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:17.040400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:17.040368    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"]
Nov 02 13:22:17.053414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:17.053384    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b23818-fnh2j"]
Nov 02 13:22:17.189184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:17.189152    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:17.407197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:17.407158    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:17.567139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:17.567098    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68" podUID="f00ea282-bd5f-4e08-8def-89fb409a4d24" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:22:18.021481 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:18.021448    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a891d4b-f123-412f-8810-ae355a07ffbd" path="/var/lib/kubelet/pods/3a891d4b-f123-412f-8810-ae355a07ffbd/volumes"
Nov 02 13:22:18.849607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:18.849568    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:20.571348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:20.571313    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:20.739829 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:20.739799    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:22.207863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:22.207818    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:23.841653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:23.841600    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:23.963242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:23.963209    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:25.430961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.430929    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:25.441471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.441444    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"
Nov 02 13:22:25.516719 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.515532    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00ea282-bd5f-4e08-8def-89fb409a4d24-kube-api-access-gjqlx\" (UniqueName: \"kubernetes.io/projected/f00ea282-bd5f-4e08-8def-89fb409a4d24-kube-api-access-gjqlx\") pod \"f00ea282-bd5f-4e08-8def-89fb409a4d24\" (UID: \"f00ea282-bd5f-4e08-8def-89fb409a4d24\") "
Nov 02 13:22:25.516719 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.515815    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f00ea282-bd5f-4e08-8def-89fb409a4d24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f00ea282-bd5f-4e08-8def-89fb409a4d24-plugins-config-volume\") pod \"f00ea282-bd5f-4e08-8def-89fb409a4d24\" (UID: \"f00ea282-bd5f-4e08-8def-89fb409a4d24\") "
Nov 02 13:22:25.517553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.517204    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00ea282-bd5f-4e08-8def-89fb409a4d24-plugins-config-volume" pod "f00ea282-bd5f-4e08-8def-89fb409a4d24" (UID: "f00ea282-bd5f-4e08-8def-89fb409a4d24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:25.535812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.535531    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00ea282-bd5f-4e08-8def-89fb409a4d24-kube-api-access-gjqlx" pod "f00ea282-bd5f-4e08-8def-89fb409a4d24" (UID: "f00ea282-bd5f-4e08-8def-89fb409a4d24"). InnerVolumeSpecName "kube-api-access-gjqlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:25.546024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.542816    5879 generic.go:356] "Generic (PLEG): container finished" podID="f00ea282-bd5f-4e08-8def-89fb409a4d24" containerID="fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe" exitCode=0
Nov 02 13:22:25.546024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.542896    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68" event={"ID":"f00ea282-bd5f-4e08-8def-89fb409a4d24","Type":"ContainerDied","Data":"fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe"}
Nov 02 13:22:25.546024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.542945    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68" event={"ID":"f00ea282-bd5f-4e08-8def-89fb409a4d24","Type":"ContainerDied","Data":"53712ad8f066a90bc2535229943d22758f3f72784e2d79e183080dc1e67dc230"}
Nov 02 13:22:25.546024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.542974    5879 scope.go:122] "RemoveContainer" containerID="fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe"
Nov 02 13:22:25.546024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.543165    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"
Nov 02 13:22:25.570132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.570109    5879 scope.go:122] "RemoveContainer" containerID="fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe"
Nov 02 13:22:25.571131 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:22:25.571102    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe\": container with ID starting with fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe not found: ID does not exist" containerID="fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe"
Nov 02 13:22:25.571219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.571143    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe"} err="failed to get container status \"fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe\": rpc error: code = NotFound desc = could not find container \"fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe\": container with ID starting with fbcecb8d8205a82e91dfbc97bce7c0c8cf9286d2e49d60c0573d4691d0a899fe not found: ID does not exist"
Nov 02 13:22:25.619643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.619593    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjqlx\" (UniqueName: \"kubernetes.io/projected/f00ea282-bd5f-4e08-8def-89fb409a4d24-kube-api-access-gjqlx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:25.619643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.619630    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f00ea282-bd5f-4e08-8def-89fb409a4d24-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:25.624590 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.624562    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"]
Nov 02 13:22:25.632095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:25.632060    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5f8899-95f48b575-rlt68"]
Nov 02 13:22:26.019311 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:26.019280    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00ea282-bd5f-4e08-8def-89fb409a4d24" path="/var/lib/kubelet/pods/f00ea282-bd5f-4e08-8def-89fb409a4d24/volumes"
Nov 02 13:22:26.952476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:26.952440    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:28.754246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:28.754153    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:30.179306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:30.179250    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:30.856183 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:30.856121    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:22:30.856663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:30.856609    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:22:31.174338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:31.174298    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:31.318464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:31.318429    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:31.925027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:31.924970    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:32.274811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:32.274718    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"]
Nov 02 13:22:32.275078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:32.275052    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj" podUID="f561b255-e7a4-4234-981e-de48e987b6f7" containerName="istio-proxy" containerID="cri-o://002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628" gracePeriod=30
Nov 02 13:22:32.288213 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:32.287869    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"]
Nov 02 13:22:32.288213 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:32.288095    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx" podUID="cf97cd15-75c5-4b58-90b0-396865600faf" containerName="istio-proxy" containerID="cri-o://c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9" gracePeriod=30
Nov 02 13:22:32.294041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:32.294017    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"]
Nov 02 13:22:32.294225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:32.294193    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz" podUID="d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" containerName="istio-proxy" containerID="cri-o://f7ae730d323a87589c9bb46160a4bdbbfb55be057ff137489c7b8cbf47800744" gracePeriod=30
Nov 02 13:22:32.352885 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:32.352853    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:33.013815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:33.013786    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:33.151247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:33.151204    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:34.806090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:34.806046    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:36.453746 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:36.453691    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:37.542879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.542786    5879 scope.go:122] "RemoveContainer" containerID="2cae1efd8435b71b880ef9c00ae5bd7a5b74daad2eda56567fd3478b25a3e272"
Nov 02 13:22:37.635957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.635559    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:22:37.665419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.665390    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:22:37.672241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.672213    5879 generic.go:356] "Generic (PLEG): container finished" podID="f561b255-e7a4-4234-981e-de48e987b6f7" containerID="002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628" exitCode=0
Nov 02 13:22:37.672381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.672253    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"
Nov 02 13:22:37.672381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.672301    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj" event={"ID":"f561b255-e7a4-4234-981e-de48e987b6f7","Type":"ContainerDied","Data":"002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628"}
Nov 02 13:22:37.672381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.672356    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj" event={"ID":"f561b255-e7a4-4234-981e-de48e987b6f7","Type":"ContainerDied","Data":"1da9bb2a8c1aea2275c4658ba6c4064d032087ae8ae2607de01495d3482727be"}
Nov 02 13:22:37.672557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.672388    5879 scope.go:122] "RemoveContainer" containerID="002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628"
Nov 02 13:22:37.674066 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.674048    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"
Nov 02 13:22:37.674194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.674086    5879 generic.go:356] "Generic (PLEG): container finished" podID="cf97cd15-75c5-4b58-90b0-396865600faf" containerID="c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9" exitCode=0
Nov 02 13:22:37.674194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.674108    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx" event={"ID":"cf97cd15-75c5-4b58-90b0-396865600faf","Type":"ContainerDied","Data":"c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9"}
Nov 02 13:22:37.674194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.674158    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx" event={"ID":"cf97cd15-75c5-4b58-90b0-396865600faf","Type":"ContainerDied","Data":"72982c31f9f684fb45fd2e3b716a0d7abdabcf179eff48bcf06110830903f461"}
Nov 02 13:22:37.676627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.676516    5879 generic.go:356] "Generic (PLEG): container finished" podID="d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" containerID="f7ae730d323a87589c9bb46160a4bdbbfb55be057ff137489c7b8cbf47800744" exitCode=0
Nov 02 13:22:37.676627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.676567    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz" event={"ID":"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8","Type":"ContainerDied","Data":"f7ae730d323a87589c9bb46160a4bdbbfb55be057ff137489c7b8cbf47800744"}
Nov 02 13:22:37.678867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.677259    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-envoy\") pod \"cf97cd15-75c5-4b58-90b0-396865600faf\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") "
Nov 02 13:22:37.678867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.677312    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-kube-api-access-p22cl\" (UniqueName: \"kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-kube-api-access-p22cl\") pod \"cf97cd15-75c5-4b58-90b0-396865600faf\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") "
Nov 02 13:22:37.678867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.677369    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-credential-socket\") pod \"cf97cd15-75c5-4b58-90b0-396865600faf\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") "
Nov 02 13:22:37.678867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.677445    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf97cd15-75c5-4b58-90b0-396865600faf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf97cd15-75c5-4b58-90b0-396865600faf-istiod-ca-cert\") pod \"cf97cd15-75c5-4b58-90b0-396865600faf\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") "
Nov 02 13:22:37.678867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.677479    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-socket\") pod \"cf97cd15-75c5-4b58-90b0-396865600faf\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") "
Nov 02 13:22:37.678867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.677596    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf97cd15-75c5-4b58-90b0-396865600faf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf97cd15-75c5-4b58-90b0-396865600faf-istio-podinfo\") pod \"cf97cd15-75c5-4b58-90b0-396865600faf\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") "
Nov 02 13:22:37.678867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.677667    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-certs\") pod \"cf97cd15-75c5-4b58-90b0-396865600faf\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") "
Nov 02 13:22:37.678867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.677762    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-data\") pod \"cf97cd15-75c5-4b58-90b0-396865600faf\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") "
Nov 02 13:22:37.678867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.677843    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-istio-token\" (UniqueName: \"kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-istio-token\") pod \"cf97cd15-75c5-4b58-90b0-396865600faf\" (UID: \"cf97cd15-75c5-4b58-90b0-396865600faf\") "
Nov 02 13:22:37.680127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.680085    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf97cd15-75c5-4b58-90b0-396865600faf-istiod-ca-cert" pod "cf97cd15-75c5-4b58-90b0-396865600faf" (UID: "cf97cd15-75c5-4b58-90b0-396865600faf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:37.680488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.680458    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-socket" pod "cf97cd15-75c5-4b58-90b0-396865600faf" (UID: "cf97cd15-75c5-4b58-90b0-396865600faf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.680488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.680474    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-credential-socket" pod "cf97cd15-75c5-4b58-90b0-396865600faf" (UID: "cf97cd15-75c5-4b58-90b0-396865600faf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.680731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.680647    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-data" pod "cf97cd15-75c5-4b58-90b0-396865600faf" (UID: "cf97cd15-75c5-4b58-90b0-396865600faf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.680731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.680685    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-certs" pod "cf97cd15-75c5-4b58-90b0-396865600faf" (UID: "cf97cd15-75c5-4b58-90b0-396865600faf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.682940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.682922    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-kube-api-access-p22cl" pod "cf97cd15-75c5-4b58-90b0-396865600faf" (UID: "cf97cd15-75c5-4b58-90b0-396865600faf"). InnerVolumeSpecName "kube-api-access-p22cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.683310 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.683289    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf97cd15-75c5-4b58-90b0-396865600faf-istio-podinfo" pod "cf97cd15-75c5-4b58-90b0-396865600faf" (UID: "cf97cd15-75c5-4b58-90b0-396865600faf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:37.683467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.683312    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-istio-token" pod "cf97cd15-75c5-4b58-90b0-396865600faf" (UID: "cf97cd15-75c5-4b58-90b0-396865600faf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.684627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.684608    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-envoy" pod "cf97cd15-75c5-4b58-90b0-396865600faf" (UID: "cf97cd15-75c5-4b58-90b0-396865600faf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.686146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.686127    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:22:37.693716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.693693    5879 scope.go:122] "RemoveContainer" containerID="002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628"
Nov 02 13:22:37.694146 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:22:37.694125    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628\": container with ID starting with 002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628 not found: ID does not exist" containerID="002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628"
Nov 02 13:22:37.694244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.694158    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628"} err="failed to get container status \"002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628\": rpc error: code = NotFound desc = could not find container \"002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628\": container with ID starting with 002123f55ee6fc3fb6447f8db3ca8c41969f8f37e6ff02c8b823dba0b4a19628 not found: ID does not exist"
Nov 02 13:22:37.694244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.694183    5879 scope.go:122] "RemoveContainer" containerID="c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9"
Nov 02 13:22:37.707278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.707252    5879 scope.go:122] "RemoveContainer" containerID="c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9"
Nov 02 13:22:37.707640 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:22:37.707611    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9\": container with ID starting with c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9 not found: ID does not exist" containerID="c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9"
Nov 02 13:22:37.707751 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.707651    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9"} err="failed to get container status \"c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9\": rpc error: code = NotFound desc = could not find container \"c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9\": container with ID starting with c28b3830d72d8473b3687189c343615c0d0a8dda23451ab94c1daa4c22bd64f9 not found: ID does not exist"
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779326    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-envoy\") pod \"f561b255-e7a4-4234-981e-de48e987b6f7\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779382    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-certs\") pod \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779474    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-data\") pod \"f561b255-e7a4-4234-981e-de48e987b6f7\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779523    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f561b255-e7a4-4234-981e-de48e987b6f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f561b255-e7a4-4234-981e-de48e987b6f7-istio-podinfo\") pod \"f561b255-e7a4-4234-981e-de48e987b6f7\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779566    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-envoy\") pod \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779638    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-socket\") pod \"f561b255-e7a4-4234-981e-de48e987b6f7\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779664    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-credential-socket\") pod \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779648    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-certs" pod "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" (UID: "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779717    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-data" pod "f561b255-e7a4-4234-981e-de48e987b6f7" (UID: "f561b255-e7a4-4234-981e-de48e987b6f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779754    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f561b255-e7a4-4234-981e-de48e987b6f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f561b255-e7a4-4234-981e-de48e987b6f7-istiod-ca-cert\") pod \"f561b255-e7a4-4234-981e-de48e987b6f7\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779801    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-credential-socket\") pod \"f561b255-e7a4-4234-981e-de48e987b6f7\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779834    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-token\" (UniqueName: \"kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-token\") pod \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779875    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-socket\") pod \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779915    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-data\") pod \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779941    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-istio-token\" (UniqueName: \"kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-istio-token\") pod \"f561b255-e7a4-4234-981e-de48e987b6f7\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") "
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779939    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-socket" pod "f561b255-e7a4-4234-981e-de48e987b6f7" (UID: "f561b255-e7a4-4234-981e-de48e987b6f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.779965    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-credential-socket" pod "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" (UID: "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.780204    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-socket" pod "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" (UID: "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.780284    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-data" pod "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" (UID: "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.780372    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f561b255-e7a4-4234-981e-de48e987b6f7-istiod-ca-cert" pod "f561b255-e7a4-4234-981e-de48e987b6f7" (UID: "f561b255-e7a4-4234-981e-de48e987b6f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:37.780482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.780449    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-credential-socket" pod "f561b255-e7a4-4234-981e-de48e987b6f7" (UID: "f561b255-e7a4-4234-981e-de48e987b6f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.781984    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-kube-api-access-zg6db\" (UniqueName: \"kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-kube-api-access-zg6db\") pod \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") "
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.782055    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-kube-api-access-45p82\" (UniqueName: \"kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-kube-api-access-45p82\") pod \"f561b255-e7a4-4234-981e-de48e987b6f7\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") "
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.782111    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-certs\") pod \"f561b255-e7a4-4234-981e-de48e987b6f7\" (UID: \"f561b255-e7a4-4234-981e-de48e987b6f7\") "
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.782225    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istiod-ca-cert\") pod \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") "
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.782251    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-podinfo\") pod \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\" (UID: \"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8\") "
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.782717    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-envoy" pod "f561b255-e7a4-4234-981e-de48e987b6f7" (UID: "f561b255-e7a4-4234-981e-de48e987b6f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.782942    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f561b255-e7a4-4234-981e-de48e987b6f7-istio-podinfo" pod "f561b255-e7a4-4234-981e-de48e987b6f7" (UID: "f561b255-e7a4-4234-981e-de48e987b6f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783068    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783091    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783104    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783111    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istiod-ca-cert" pod "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" (UID: "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783117    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p22cl\" (UniqueName: \"kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-kube-api-access-p22cl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783154    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783173    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783187    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf97cd15-75c5-4b58-90b0-396865600faf-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783202    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783214    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783228    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783242    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f561b255-e7a4-4234-981e-de48e987b6f7-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783350    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf97cd15-75c5-4b58-90b0-396865600faf-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783363    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783378    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783391    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783395    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-certs" pod "f561b255-e7a4-4234-981e-de48e987b6f7" (UID: "f561b255-e7a4-4234-981e-de48e987b6f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783407    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783433    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf97cd15-75c5-4b58-90b0-396865600faf-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.783471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.783449    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf97cd15-75c5-4b58-90b0-396865600faf-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.785054 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.785023    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-podinfo" pod "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" (UID: "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:22:37.790494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.790423    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-kube-api-access-zg6db" pod "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" (UID: "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8"). InnerVolumeSpecName "kube-api-access-zg6db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.790494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.790462    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-istio-token" pod "f561b255-e7a4-4234-981e-de48e987b6f7" (UID: "f561b255-e7a4-4234-981e-de48e987b6f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.791893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.791830    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-envoy" pod "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" (UID: "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:22:37.791893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.791856    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-token" pod "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" (UID: "d6f8c18c-bc7b-4475-a7b8-153e0b7648a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.792178 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.792150    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-kube-api-access-45p82" pod "f561b255-e7a4-4234-981e-de48e987b6f7" (UID: "f561b255-e7a4-4234-981e-de48e987b6f7"). InnerVolumeSpecName "kube-api-access-45p82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:22:37.884863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.884825    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.884863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.884864    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.885116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.884878    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg6db\" (UniqueName: \"kubernetes.io/projected/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-kube-api-access-zg6db\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.885116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.884893    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45p82\" (UniqueName: \"kubernetes.io/projected/f561b255-e7a4-4234-981e-de48e987b6f7-kube-api-access-45p82\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.885116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.884908    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f561b255-e7a4-4234-981e-de48e987b6f7-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.885116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.884921    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.885116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.884934    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.885116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.884948    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f561b255-e7a4-4234-981e-de48e987b6f7-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:37.885116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:37.884960    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:22:38.040083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:38.040044    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"]
Nov 02 13:22:38.048413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:38.043812    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-srqbx"]
Nov 02 13:22:38.077559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:38.077524    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"]
Nov 02 13:22:38.083484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:38.083458    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-b4vsj"]
Nov 02 13:22:38.173731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:38.173597    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:38.688945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:38.688911    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"
Nov 02 13:22:38.688945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:38.688928    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz" event={"ID":"d6f8c18c-bc7b-4475-a7b8-153e0b7648a8","Type":"ContainerDied","Data":"62ee533cf5f1c48bb212f20b79bff84351b02cd59c0637686d01882285e41954"}
Nov 02 13:22:38.689524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:38.689054    5879 scope.go:122] "RemoveContainer" containerID="f7ae730d323a87589c9bb46160a4bdbbfb55be057ff137489c7b8cbf47800744"
Nov 02 13:22:38.749521 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:38.749483    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"]
Nov 02 13:22:38.762981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:38.762952    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6563ed-openshift-default-bbf46f5bf-zqjnz"]
Nov 02 13:22:39.615306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:39.615271    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"]
Nov 02 13:22:39.760683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:39.760644    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:39.960986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:39.960901    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:40.020507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:40.020465    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf97cd15-75c5-4b58-90b0-396865600faf" path="/var/lib/kubelet/pods/cf97cd15-75c5-4b58-90b0-396865600faf/volumes"
Nov 02 13:22:40.021139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:40.021117    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" path="/var/lib/kubelet/pods/d6f8c18c-bc7b-4475-a7b8-153e0b7648a8/volumes"
Nov 02 13:22:40.021633 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:40.021618    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f561b255-e7a4-4234-981e-de48e987b6f7" path="/var/lib/kubelet/pods/f561b255-e7a4-4234-981e-de48e987b6f7/volumes"
Nov 02 13:22:40.265605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:40.265510    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:40.394064 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:40.394033    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:41.465682 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:41.465639    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:42.835568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:42.835527    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:43.302297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.302226    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"]
Nov 02 13:22:43.303750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.303720    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c77bd7e8-4b52-427b-90c5-ad7460330a8a" containerName="istio-proxy"
Nov 02 13:22:43.303954 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.303936    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c77bd7e8-4b52-427b-90c5-ad7460330a8a" containerName="istio-proxy"
Nov 02 13:22:43.304096 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.304081    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cf97cd15-75c5-4b58-90b0-396865600faf" containerName="istio-proxy"
Nov 02 13:22:43.304179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.304169    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf97cd15-75c5-4b58-90b0-396865600faf" containerName="istio-proxy"
Nov 02 13:22:43.304278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.304268    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e168fd9f-aa07-4650-b616-37ee7b92ee77" containerName="main"
Nov 02 13:22:43.304382 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.304362    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e168fd9f-aa07-4650-b616-37ee7b92ee77" containerName="main"
Nov 02 13:22:43.304462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.304399    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f00ea282-bd5f-4e08-8def-89fb409a4d24" containerName="payload-processing"
Nov 02 13:22:43.304462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.304408    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00ea282-bd5f-4e08-8def-89fb409a4d24" containerName="payload-processing"
Nov 02 13:22:43.304462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.304435    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" containerName="istio-proxy"
Nov 02 13:22:43.304462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.304443    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" containerName="istio-proxy"
Nov 02 13:22:43.304662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.304477    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f561b255-e7a4-4234-981e-de48e987b6f7" containerName="istio-proxy"
Nov 02 13:22:43.304662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.304486    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f561b255-e7a4-4234-981e-de48e987b6f7" containerName="istio-proxy"
Nov 02 13:22:43.306628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.306606    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c77bd7e8-4b52-427b-90c5-ad7460330a8a" containerName="istio-proxy"
Nov 02 13:22:43.307379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.307360    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf97cd15-75c5-4b58-90b0-396865600faf" containerName="istio-proxy"
Nov 02 13:22:43.307511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.307496    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e168fd9f-aa07-4650-b616-37ee7b92ee77" containerName="main"
Nov 02 13:22:43.307585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.307521    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00ea282-bd5f-4e08-8def-89fb409a4d24" containerName="payload-processing"
Nov 02 13:22:43.307585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.307536    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f8c18c-bc7b-4475-a7b8-153e0b7648a8" containerName="istio-proxy"
Nov 02 13:22:43.307585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.307551    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f561b255-e7a4-4234-981e-de48e987b6f7" containerName="istio-proxy"
Nov 02 13:22:43.318403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.318366    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.343560 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.343524    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"]
Nov 02 13:22:43.380726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.380681    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.380950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.380803    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.380950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.380870    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.381167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.381125    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.381230 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.381197    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.381300 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.381236    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.381553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.381530    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sc5r\" (UniqueName: \"kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-kube-api-access-7sc5r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.381697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.381576    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.381766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.381703    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.473766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.473727    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:43.483361 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.483328    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.483544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.483390    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.483544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.483423    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.483544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.483517    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sc5r\" (UniqueName: \"kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-kube-api-access-7sc5r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.483780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.483546    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.483780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.483586    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.483780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.483734    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.483780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.483772    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.484238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.483825    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.484238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.483916    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.484394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.484263    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.484394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.484273    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.484538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.484514    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.484590 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.484523    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.486616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.486589    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.486730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.486634    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.510189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.510156    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.510430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.510376    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sc5r\" (UniqueName: \"kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-kube-api-access-7sc5r\") pod \"maas-default-gateway-openshift-default-5bc884dd87-4ncwb\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.653419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.653383    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:43.984967 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:22:43.984937    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cfe2d10_a2c9_4145_abe6_3a9e66b1e9cc.slice/crio-a4d9a75deb2d0fcb9ba68ab7943e928bc488709a3594c3816f331f0c2acbc8d9 WatchSource:0}: Error finding container a4d9a75deb2d0fcb9ba68ab7943e928bc488709a3594c3816f331f0c2acbc8d9: Status 404 returned error can't find the container with id a4d9a75deb2d0fcb9ba68ab7943e928bc488709a3594c3816f331f0c2acbc8d9
Nov 02 13:22:43.986213 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.986186    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"]
Nov 02 13:22:43.988183 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.988151    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:22:43.988270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.988215    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:22:43.988270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:43.988243    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:22:44.616084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:44.616040    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn" podUID="3169f74e-ae11-4f79-80bd-5e400884ac03" containerName="payload-pre-processing" containerID="cri-o://0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc" gracePeriod=25
Nov 02 13:22:44.815280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:44.815244    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" event={"ID":"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc","Type":"ContainerStarted","Data":"1a7ef25a8ee527b72dc999798796907b2c3a0fd8a38eda96255ec0688400aead"}
Nov 02 13:22:44.815484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:44.815304    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" event={"ID":"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc","Type":"ContainerStarted","Data":"a4d9a75deb2d0fcb9ba68ab7943e928bc488709a3594c3816f331f0c2acbc8d9"}
Nov 02 13:22:44.837790 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:22:44.837754    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:22:45.301018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:45.296419    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:45.653852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:45.653816    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:45.655936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:45.655911    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4ncwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body=
Nov 02 13:22:45.656108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:45.656020    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" podUID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused"
Nov 02 13:22:46.654685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:46.654648    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4ncwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body=
Nov 02 13:22:46.655164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:46.654712    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" podUID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused"
Nov 02 13:22:47.005076 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:47.004459    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:47.155561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:47.155523    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:47.654369 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:47.654331    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4ncwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body=
Nov 02 13:22:47.654671 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:47.654384    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" podUID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused"
Nov 02 13:22:47.813538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:47.813503    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:48.212711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:48.212676    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:48.654552 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:48.654522    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4ncwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body=
Nov 02 13:22:48.654726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:48.654572    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" podUID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused"
Nov 02 13:22:48.851615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:48.851575    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:49.222234 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:49.222194    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:49.375907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:49.375871    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:49.654412 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:49.654373    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4ncwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body=
Nov 02 13:22:49.654599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:49.654429    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" podUID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused"
Nov 02 13:22:50.106082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:50.102332    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:50.473886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:50.473840    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn" podUID="3169f74e-ae11-4f79-80bd-5e400884ac03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:22:50.654443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:50.654389    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4ncwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body=
Nov 02 13:22:50.654671 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:50.654466    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" podUID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused"
Nov 02 13:22:50.691164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:50.691126    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:51.654125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:51.654092    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-4ncwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused" start-of-body=
Nov 02 13:22:51.654548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:51.654149    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" podUID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.93:15021/healthz/ready\": dial tcp 10.134.1.93:15021: connect: connection refused"
Nov 02 13:22:52.450435 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:52.450394    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:52.824631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:52.824542    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:52.825398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:52.825372    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:52.826169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:52.826148    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:22:52.891253 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:52.891192    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" podStartSLOduration=9.89117344 podStartE2EDuration="9.89117344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:22:44.898335989 +0000 UTC m=+3311.005038852" watchObservedRunningTime="2026-11-02 13:22:52.89117344 +0000 UTC m=+3318.997876294"
Nov 02 13:22:54.045114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:54.045083    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:55.652688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:55.652652    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:57.293583 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:57.293546    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:22:58.785553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:22:58.785521    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:00.215638 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:00.215598    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:00.473307 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:00.473194    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn" podUID="3169f74e-ae11-4f79-80bd-5e400884ac03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:23:00.973199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:00.973166    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:01.647237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:01.647203    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:01.861077 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:01.861040    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:02.384536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:02.384503    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:02.659013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:02.658905    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:02.832044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:02.831972    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:03.760646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:03.760611    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:03.891232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:03.891198    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:04.354077 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:04.354044    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:04.912290 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:04.912260    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"
Nov 02 13:23:04.994475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:04.994429    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3169f74e-ae11-4f79-80bd-5e400884ac03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3169f74e-ae11-4f79-80bd-5e400884ac03-plugins-config-volume\") pod \"3169f74e-ae11-4f79-80bd-5e400884ac03\" (UID: \"3169f74e-ae11-4f79-80bd-5e400884ac03\") "
Nov 02 13:23:04.994694 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:04.994512    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3169f74e-ae11-4f79-80bd-5e400884ac03-kube-api-access-6rsjq\" (UniqueName: \"kubernetes.io/projected/3169f74e-ae11-4f79-80bd-5e400884ac03-kube-api-access-6rsjq\") pod \"3169f74e-ae11-4f79-80bd-5e400884ac03\" (UID: \"3169f74e-ae11-4f79-80bd-5e400884ac03\") "
Nov 02 13:23:04.994948 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:04.994918    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3169f74e-ae11-4f79-80bd-5e400884ac03-plugins-config-volume" pod "3169f74e-ae11-4f79-80bd-5e400884ac03" (UID: "3169f74e-ae11-4f79-80bd-5e400884ac03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:23:04.995185 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:04.995163    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3169f74e-ae11-4f79-80bd-5e400884ac03-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:04.997718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:04.997695    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3169f74e-ae11-4f79-80bd-5e400884ac03-kube-api-access-6rsjq" pod "3169f74e-ae11-4f79-80bd-5e400884ac03" (UID: "3169f74e-ae11-4f79-80bd-5e400884ac03"). InnerVolumeSpecName "kube-api-access-6rsjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:05.009203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:05.009074    5879 generic.go:356] "Generic (PLEG): container finished" podID="3169f74e-ae11-4f79-80bd-5e400884ac03" containerID="0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc" exitCode=0
Nov 02 13:23:05.009203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:05.009104    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn" event={"ID":"3169f74e-ae11-4f79-80bd-5e400884ac03","Type":"ContainerDied","Data":"0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc"}
Nov 02 13:23:05.009203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:05.009137    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"
Nov 02 13:23:05.009203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:05.009156    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn" event={"ID":"3169f74e-ae11-4f79-80bd-5e400884ac03","Type":"ContainerDied","Data":"74dfd75792b60e70e352b7bedcd9bd465f34c6c5d40c80d7c0b32d87b866b423"}
Nov 02 13:23:05.009203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:05.009182    5879 scope.go:122] "RemoveContainer" containerID="0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc"
Nov 02 13:23:05.025610 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:05.025588    5879 scope.go:122] "RemoveContainer" containerID="0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc"
Nov 02 13:23:05.025927 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:23:05.025906    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc\": container with ID starting with 0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc not found: ID does not exist" containerID="0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc"
Nov 02 13:23:05.025978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:05.025936    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc"} err="failed to get container status \"0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc\": rpc error: code = NotFound desc = could not find container \"0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc\": container with ID starting with 0df6c1bea365bae5969da5470de98175eed642b2210ea476b2b72584c59496fc not found: ID does not exist"
Nov 02 13:23:05.071976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:05.071937    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"]
Nov 02 13:23:05.076747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:05.076711    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-361c3a-8648dbb499-fp9wn"]
Nov 02 13:23:05.096286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:05.096250    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rsjq\" (UniqueName: \"kubernetes.io/projected/3169f74e-ae11-4f79-80bd-5e400884ac03-kube-api-access-6rsjq\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:05.928426 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:05.928388    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:06.019216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:06.019182    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3169f74e-ae11-4f79-80bd-5e400884ac03" path="/var/lib/kubelet/pods/3169f74e-ae11-4f79-80bd-5e400884ac03/volumes"
Nov 02 13:23:07.469584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:07.469545    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:08.355063 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:08.355033    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:09.166491 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:09.166460    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:10.751916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:10.751879    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:12.314259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:12.314196    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:14.109198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:14.109112    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:14.823469 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:23:14.823421    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:23:15.663524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:15.663495    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:16.220407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:16.220365    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:17.305524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:17.305486    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:18.966896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:18.966861    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:19.446264 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:19.446231    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"]
Nov 02 13:23:19.446477 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:19.446455    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp" podUID="cf891163-0b6a-4951-8730-04154ea0b7e9" containerName="istio-proxy" containerID="cri-o://a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a" gracePeriod=30
Nov 02 13:23:20.104525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:20.104492    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:20.671091 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:20.671053    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:22.264054 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:22.264019    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:22.869146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:22.869107    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:23.976791 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:23.976758    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:24.769655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.769624    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:23:24.799277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.799239    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-socket\") pod \"cf891163-0b6a-4951-8730-04154ea0b7e9\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") "
Nov 02 13:23:24.799494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.799310    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-kube-api-access-pbtb4\" (UniqueName: \"kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-kube-api-access-pbtb4\") pod \"cf891163-0b6a-4951-8730-04154ea0b7e9\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") "
Nov 02 13:23:24.799494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.799366    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-envoy\") pod \"cf891163-0b6a-4951-8730-04154ea0b7e9\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") "
Nov 02 13:23:24.799494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.799437    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-certs\") pod \"cf891163-0b6a-4951-8730-04154ea0b7e9\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") "
Nov 02 13:23:24.799494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.799470    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-podinfo\") pod \"cf891163-0b6a-4951-8730-04154ea0b7e9\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") "
Nov 02 13:23:24.799836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.799498    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-token\" (UniqueName: \"kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-token\") pod \"cf891163-0b6a-4951-8730-04154ea0b7e9\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") "
Nov 02 13:23:24.799836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.799534    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-credential-socket\") pod \"cf891163-0b6a-4951-8730-04154ea0b7e9\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") "
Nov 02 13:23:24.799836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.799603    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-socket" pod "cf891163-0b6a-4951-8730-04154ea0b7e9" (UID: "cf891163-0b6a-4951-8730-04154ea0b7e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:24.799836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.799698    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf891163-0b6a-4951-8730-04154ea0b7e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf891163-0b6a-4951-8730-04154ea0b7e9-istiod-ca-cert\") pod \"cf891163-0b6a-4951-8730-04154ea0b7e9\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") "
Nov 02 13:23:24.799836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.799813    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-data\") pod \"cf891163-0b6a-4951-8730-04154ea0b7e9\" (UID: \"cf891163-0b6a-4951-8730-04154ea0b7e9\") "
Nov 02 13:23:24.800158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.799931    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-credential-socket" pod "cf891163-0b6a-4951-8730-04154ea0b7e9" (UID: "cf891163-0b6a-4951-8730-04154ea0b7e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:24.800158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.800109    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf891163-0b6a-4951-8730-04154ea0b7e9-istiod-ca-cert" pod "cf891163-0b6a-4951-8730-04154ea0b7e9" (UID: "cf891163-0b6a-4951-8730-04154ea0b7e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:23:24.800283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.800249    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-certs" pod "cf891163-0b6a-4951-8730-04154ea0b7e9" (UID: "cf891163-0b6a-4951-8730-04154ea0b7e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:24.800283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.800265    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-data" pod "cf891163-0b6a-4951-8730-04154ea0b7e9" (UID: "cf891163-0b6a-4951-8730-04154ea0b7e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:24.800778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.800756    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:24.800871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.800782    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf891163-0b6a-4951-8730-04154ea0b7e9-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:24.800871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.800796    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:24.800871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.800807    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:24.800871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.800820    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:24.802113 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.802087    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-podinfo" pod "cf891163-0b6a-4951-8730-04154ea0b7e9" (UID: "cf891163-0b6a-4951-8730-04154ea0b7e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:23:24.802430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.802405    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-token" pod "cf891163-0b6a-4951-8730-04154ea0b7e9" (UID: "cf891163-0b6a-4951-8730-04154ea0b7e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:24.802537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.802431    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-envoy" pod "cf891163-0b6a-4951-8730-04154ea0b7e9" (UID: "cf891163-0b6a-4951-8730-04154ea0b7e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:23:24.802537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.802434    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-kube-api-access-pbtb4" pod "cf891163-0b6a-4951-8730-04154ea0b7e9" (UID: "cf891163-0b6a-4951-8730-04154ea0b7e9"). InnerVolumeSpecName "kube-api-access-pbtb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:24.902360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.902322    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbtb4\" (UniqueName: \"kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-kube-api-access-pbtb4\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:24.902360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.902353    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:24.902360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.902364    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:24.902360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:24.902372    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf891163-0b6a-4951-8730-04154ea0b7e9-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:25.182082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:25.182042    5879 generic.go:356] "Generic (PLEG): container finished" podID="cf891163-0b6a-4951-8730-04154ea0b7e9" containerID="a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a" exitCode=0
Nov 02 13:23:25.182568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:25.182159    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"
Nov 02 13:23:25.182568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:25.182169    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp" event={"ID":"cf891163-0b6a-4951-8730-04154ea0b7e9","Type":"ContainerDied","Data":"a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a"}
Nov 02 13:23:25.182568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:25.182250    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp" event={"ID":"cf891163-0b6a-4951-8730-04154ea0b7e9","Type":"ContainerDied","Data":"5dc0cca9da57c6c733d2ec676ce0b2f4dce6fe8b2788426a060d8a081e2e87bc"}
Nov 02 13:23:25.182568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:25.182280    5879 scope.go:122] "RemoveContainer" containerID="a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a"
Nov 02 13:23:25.199402 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:25.199302    5879 scope.go:122] "RemoveContainer" containerID="a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a"
Nov 02 13:23:25.199800 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:23:25.199776    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a\": container with ID starting with a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a not found: ID does not exist" containerID="a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a"
Nov 02 13:23:25.199899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:25.199807    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a"} err="failed to get container status \"a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a\": rpc error: code = NotFound desc = could not find container \"a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a\": container with ID starting with a28f12e6152c1d313361f50d1a9b849a03362c3cfb3e84beb269665968f9f98a not found: ID does not exist"
Nov 02 13:23:25.242612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:25.242580    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"]
Nov 02 13:23:25.256805 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:25.256769    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-90cda9-openshift-default-57f8b6f957-p9mqp"]
Nov 02 13:23:25.664364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:25.664310    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:26.019534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:26.019443    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf891163-0b6a-4951-8730-04154ea0b7e9" path="/var/lib/kubelet/pods/cf891163-0b6a-4951-8730-04154ea0b7e9/volumes"
Nov 02 13:23:26.419210 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:26.419177    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:27.235780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:27.235725    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:28.780678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:28.780646    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:30.355405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.355366    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"]
Nov 02 13:23:30.356204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.356186    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cf891163-0b6a-4951-8730-04154ea0b7e9" containerName="istio-proxy"
Nov 02 13:23:30.356204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.356204    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf891163-0b6a-4951-8730-04154ea0b7e9" containerName="istio-proxy"
Nov 02 13:23:30.356314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.356271    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3169f74e-ae11-4f79-80bd-5e400884ac03" containerName="payload-pre-processing"
Nov 02 13:23:30.356314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.356277    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3169f74e-ae11-4f79-80bd-5e400884ac03" containerName="payload-pre-processing"
Nov 02 13:23:30.356410 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.356400    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3169f74e-ae11-4f79-80bd-5e400884ac03" containerName="payload-pre-processing"
Nov 02 13:23:30.356451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.356415    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf891163-0b6a-4951-8730-04154ea0b7e9" containerName="istio-proxy"
Nov 02 13:23:30.362400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.362376    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:30.383769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.383726    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"]
Nov 02 13:23:30.391734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.391704    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:23:30.392787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.392758    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:23:30.474644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.474609    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:30.486844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.486779    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-562sl\" (UniqueName: \"kubernetes.io/projected/05cd217b-d890-4bdf-a282-5deb6decb94d-kube-api-access-562sl\") pod \"maas-api-revoke-sub-d25d2853-xplwk\" (UID: \"05cd217b-d890-4bdf-a282-5deb6decb94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:30.487216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.486896    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d25d2853-xplwk\" (UID: \"05cd217b-d890-4bdf-a282-5deb6decb94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:30.487216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.486952    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d25d2853-xplwk\" (UID: \"05cd217b-d890-4bdf-a282-5deb6decb94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:30.588462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.588424    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-562sl\" (UniqueName: \"kubernetes.io/projected/05cd217b-d890-4bdf-a282-5deb6decb94d-kube-api-access-562sl\") pod \"maas-api-revoke-sub-d25d2853-xplwk\" (UID: \"05cd217b-d890-4bdf-a282-5deb6decb94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:30.588681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.588478    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d25d2853-xplwk\" (UID: \"05cd217b-d890-4bdf-a282-5deb6decb94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:30.588681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.588509    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d25d2853-xplwk\" (UID: \"05cd217b-d890-4bdf-a282-5deb6decb94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:30.589378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.589304    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d25d2853-xplwk\" (UID: \"05cd217b-d890-4bdf-a282-5deb6decb94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:30.591172 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.591151    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d25d2853-xplwk\" (UID: \"05cd217b-d890-4bdf-a282-5deb6decb94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:30.614431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.614390    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-562sl\" (UniqueName: \"kubernetes.io/projected/05cd217b-d890-4bdf-a282-5deb6decb94d-kube-api-access-562sl\") pod \"maas-api-revoke-sub-d25d2853-xplwk\" (UID: \"05cd217b-d890-4bdf-a282-5deb6decb94d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:30.677816 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.677771    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:30.998077 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:23:30.998042    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05cd217b_d890_4bdf_a282_5deb6decb94d.slice/crio-22f7c1fc757c8da609ef15676f480d41bd1e16320d96ba22c3e5b9e4100e9230 WatchSource:0}: Error finding container 22f7c1fc757c8da609ef15676f480d41bd1e16320d96ba22c3e5b9e4100e9230: Status 404 returned error can't find the container with id 22f7c1fc757c8da609ef15676f480d41bd1e16320d96ba22c3e5b9e4100e9230
Nov 02 13:23:30.999379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:30.999350    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"]
Nov 02 13:23:31.234356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:31.234326    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk" event={"ID":"05cd217b-d890-4bdf-a282-5deb6decb94d","Type":"ContainerStarted","Data":"22f7c1fc757c8da609ef15676f480d41bd1e16320d96ba22c3e5b9e4100e9230"}
Nov 02 13:23:32.244490 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:32.244454    5879 generic.go:356] "Generic (PLEG): container finished" podID="05cd217b-d890-4bdf-a282-5deb6decb94d" containerID="cd49469781c59dea57f3e474f9579980bf33450b5b9d8678340152b7d6650d20" exitCode=0
Nov 02 13:23:32.244882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:32.244500    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk" event={"ID":"05cd217b-d890-4bdf-a282-5deb6decb94d","Type":"ContainerDied","Data":"cd49469781c59dea57f3e474f9579980bf33450b5b9d8678340152b7d6650d20"}
Nov 02 13:23:32.324822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:32.324657    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:33.418406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:33.418380    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:33.534590 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:33.534495    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-xks-ca\") pod \"05cd217b-d890-4bdf-a282-5deb6decb94d\" (UID: \"05cd217b-d890-4bdf-a282-5deb6decb94d\") "
Nov 02 13:23:33.534804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:33.534654    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-openshift-ca\") pod \"05cd217b-d890-4bdf-a282-5deb6decb94d\" (UID: \"05cd217b-d890-4bdf-a282-5deb6decb94d\") "
Nov 02 13:23:33.534804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:33.534719    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05cd217b-d890-4bdf-a282-5deb6decb94d-kube-api-access-562sl\" (UniqueName: \"kubernetes.io/projected/05cd217b-d890-4bdf-a282-5deb6decb94d-kube-api-access-562sl\") pod \"05cd217b-d890-4bdf-a282-5deb6decb94d\" (UID: \"05cd217b-d890-4bdf-a282-5deb6decb94d\") "
Nov 02 13:23:33.535090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:33.535052    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-openshift-ca" pod "05cd217b-d890-4bdf-a282-5deb6decb94d" (UID: "05cd217b-d890-4bdf-a282-5deb6decb94d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:23:33.535367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:33.535351    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:33.537081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:33.537048    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-xks-ca" pod "05cd217b-d890-4bdf-a282-5deb6decb94d" (UID: "05cd217b-d890-4bdf-a282-5deb6decb94d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:23:33.537185 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:33.537104    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05cd217b-d890-4bdf-a282-5deb6decb94d-kube-api-access-562sl" pod "05cd217b-d890-4bdf-a282-5deb6decb94d" (UID: "05cd217b-d890-4bdf-a282-5deb6decb94d"). InnerVolumeSpecName "kube-api-access-562sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:23:33.637242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:33.637193    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-562sl\" (UniqueName: \"kubernetes.io/projected/05cd217b-d890-4bdf-a282-5deb6decb94d-kube-api-access-562sl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:33.637242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:33.637232    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05cd217b-d890-4bdf-a282-5deb6decb94d-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:23:33.931173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:33.931126    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:34.263022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:34.262912    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"
Nov 02 13:23:34.263022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:34.262924    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk" event={"ID":"05cd217b-d890-4bdf-a282-5deb6decb94d","Type":"ContainerDied","Data":"22f7c1fc757c8da609ef15676f480d41bd1e16320d96ba22c3e5b9e4100e9230"}
Nov 02 13:23:34.263022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:34.262968    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22f7c1fc757c8da609ef15676f480d41bd1e16320d96ba22c3e5b9e4100e9230"
Nov 02 13:23:35.556008 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:35.555958    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:37.168885 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:37.168799    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:37.629196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:37.629164    5879 scope.go:122] "RemoveContainer" containerID="de9532f383b97894c658f4099b7bf506b76b40f3da0d684b1b4fa57df24c5882"
Nov 02 13:23:37.641377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:37.641349    5879 scope.go:122] "RemoveContainer" containerID="ef7e9e89f709027983ce0eba291fa829d60eefc191c8231e69c07956e22df6ab"
Nov 02 13:23:37.653072 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:37.653044    5879 scope.go:122] "RemoveContainer" containerID="a4cbcdf1f44951272f6fbeb94f1b30c317063b603dd3c75c4f220e5759e7d064"
Nov 02 13:23:38.830052 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:38.829985    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:40.391681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:40.391642    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:40.529351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:40.529316    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:40.759499 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:40.759417    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:42.211803 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:42.211764    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:43.733054 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:43.732972    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:44.823425 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:23:44.823390    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:23:45.248189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:45.248137    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:46.965111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:46.965076    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:47.184199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:47.184164    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:48.643295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:48.643230    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:50.294743 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:50.294706    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:51.962458 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:51.962412    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:53.488331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:53.488290    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:55.058588 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:55.058552    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:56.849514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:56.849476    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:23:58.753335 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:23:58.753295    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:00.355833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:00.355799    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:02.059422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:02.059374    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:03.740352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:03.740312    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:05.034831 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:05.034793    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"]
Nov 02 13:24:05.049871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:05.049833    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4dea6e7a-zrlsg"]
Nov 02 13:24:05.390775 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:05.390721    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:06.019171 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:06.019131    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4068480-4ce9-4956-ab43-22166dfcccc1" path="/var/lib/kubelet/pods/d4068480-4ce9-4956-ab43-22166dfcccc1/volumes"
Nov 02 13:24:07.070352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:07.070303    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:08.651575 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:08.651531    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:10.285634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:10.285597    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:11.950644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:11.950596    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:13.679512 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:13.679469    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:14.823885 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:24:14.823846    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:24:15.160270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:15.160235    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:16.032025 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:16.031982    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"]
Nov 02 13:24:16.045869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:16.045838    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e73f12ba-25jlq"]
Nov 02 13:24:16.748387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:16.748353    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:18.019234 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:18.019199    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25175e92-cbb7-46b6-b261-190631c4f98d" path="/var/lib/kubelet/pods/25175e92-cbb7-46b6-b261-190631c4f98d/volumes"
Nov 02 13:24:18.354709 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:18.354594    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:19.972449 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:19.972411    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:21.591388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:21.591337    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:23.258838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:23.258802    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:24.943527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:24.943478    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:26.478290 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:26.478256    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:28.054155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:28.054116    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:29.821845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:29.821809    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:31.344301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:31.344259    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:33.046598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:33.046540    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:34.642792 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:34.642745    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:36.260140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:36.260100    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:37.847161 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:37.847127    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:39.359747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:39.359706    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:40.512599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:40.512564    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:40.971801 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:40.971769    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:42.558538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:42.558500    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:44.099165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:44.099071    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:44.823840 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:24:44.823797    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:24:45.745228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:45.745184    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:46.677511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:46.677475    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:46.813338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:46.813299    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:46.959569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:46.959478    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:47.580758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:47.580725    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:47.870185 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:47.870151    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:48.012432 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:48.012399    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:48.188430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:48.188347    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:49.620830 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:49.620794    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:49.871310 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:49.871222    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"]
Nov 02 13:24:49.872047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:49.872031    5879 cpu_manager.go:414] "containerMap: removing container" podUID="05cd217b-d890-4bdf-a282-5deb6decb94d" containerName="revoke-keys"
Nov 02 13:24:49.872110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:49.872049    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05cd217b-d890-4bdf-a282-5deb6decb94d" containerName="revoke-keys"
Nov 02 13:24:49.872222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:49.872212    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="05cd217b-d890-4bdf-a282-5deb6decb94d" containerName="revoke-keys"
Nov 02 13:24:49.875637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:49.875618    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:49.897467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:49.897438    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"]
Nov 02 13:24:49.906682 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:49.906653    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:24:49.906973 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:49.906952    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:24:49.952500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:49.952463    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-162d65ca-hnpzc\" (UID: \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:49.952690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:49.952584    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-162d65ca-hnpzc\" (UID: \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:49.952690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:49.952616    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdf6z\" (UniqueName: \"kubernetes.io/projected/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-kube-api-access-bdf6z\") pod \"maas-api-revoke-sub-162d65ca-hnpzc\" (UID: \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:50.054312 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:50.054276    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-162d65ca-hnpzc\" (UID: \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:50.054546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:50.054391    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-162d65ca-hnpzc\" (UID: \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:50.054546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:50.054436    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdf6z\" (UniqueName: \"kubernetes.io/projected/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-kube-api-access-bdf6z\") pod \"maas-api-revoke-sub-162d65ca-hnpzc\" (UID: \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:50.055022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:50.054973    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-162d65ca-hnpzc\" (UID: \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:50.057143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:50.057119    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-162d65ca-hnpzc\" (UID: \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:50.080157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:50.080125    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdf6z\" (UniqueName: \"kubernetes.io/projected/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-kube-api-access-bdf6z\") pod \"maas-api-revoke-sub-162d65ca-hnpzc\" (UID: \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:50.191618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:50.191527    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:50.397101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:50.397068    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:50.502496 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:24:50.502460    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050c1f0a_55d3_4e78_a9f5_51b0652bc53c.slice/crio-e8ebef80dbd7c27e2cb7d4f9214ce40f613941a67e6dcb197a38eea55714a839 WatchSource:0}: Error finding container e8ebef80dbd7c27e2cb7d4f9214ce40f613941a67e6dcb197a38eea55714a839: Status 404 returned error can't find the container with id e8ebef80dbd7c27e2cb7d4f9214ce40f613941a67e6dcb197a38eea55714a839
Nov 02 13:24:50.503164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:50.503133    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"]
Nov 02 13:24:50.860922 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:50.860810    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:50.998017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:50.992907    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc" event={"ID":"050c1f0a-55d3-4e78-a9f5-51b0652bc53c","Type":"ContainerStarted","Data":"e8ebef80dbd7c27e2cb7d4f9214ce40f613941a67e6dcb197a38eea55714a839"}
Nov 02 13:24:51.308471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:51.308428    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:51.448306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:51.448270    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:52.002865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:52.002822    5879 generic.go:356] "Generic (PLEG): container finished" podID="050c1f0a-55d3-4e78-a9f5-51b0652bc53c" containerID="ceefdf6fe9acf7037df57f11a74375e5fd78858689a3ef48e3aec8e667c46c07" exitCode=0
Nov 02 13:24:52.003316 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:52.002885    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc" event={"ID":"050c1f0a-55d3-4e78-a9f5-51b0652bc53c","Type":"ContainerDied","Data":"ceefdf6fe9acf7037df57f11a74375e5fd78858689a3ef48e3aec8e667c46c07"}
Nov 02 13:24:53.020421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:53.020380    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:53.210305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:53.210280    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:53.317789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:53.317717    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-xks-ca\") pod \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\" (UID: \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\") "
Nov 02 13:24:53.317789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:53.317790    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-kube-api-access-bdf6z\" (UniqueName: \"kubernetes.io/projected/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-kube-api-access-bdf6z\") pod \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\" (UID: \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\") "
Nov 02 13:24:53.318064 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:53.317834    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-openshift-ca\") pod \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\" (UID: \"050c1f0a-55d3-4e78-a9f5-51b0652bc53c\") "
Nov 02 13:24:53.318247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:53.318220    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-openshift-ca" pod "050c1f0a-55d3-4e78-a9f5-51b0652bc53c" (UID: "050c1f0a-55d3-4e78-a9f5-51b0652bc53c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:24:53.318558 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:53.318520    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:24:53.319932 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:53.319909    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-xks-ca" pod "050c1f0a-55d3-4e78-a9f5-51b0652bc53c" (UID: "050c1f0a-55d3-4e78-a9f5-51b0652bc53c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:24:53.320081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:53.320059    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-kube-api-access-bdf6z" pod "050c1f0a-55d3-4e78-a9f5-51b0652bc53c" (UID: "050c1f0a-55d3-4e78-a9f5-51b0652bc53c"). InnerVolumeSpecName "kube-api-access-bdf6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:24:53.419917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:53.419876    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:24:53.419917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:53.419906    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdf6z\" (UniqueName: \"kubernetes.io/projected/050c1f0a-55d3-4e78-a9f5-51b0652bc53c-kube-api-access-bdf6z\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:24:54.026251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:54.026220    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"
Nov 02 13:24:54.026748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:54.026266    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc" event={"ID":"050c1f0a-55d3-4e78-a9f5-51b0652bc53c","Type":"ContainerDied","Data":"e8ebef80dbd7c27e2cb7d4f9214ce40f613941a67e6dcb197a38eea55714a839"}
Nov 02 13:24:54.026748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:54.026329    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8ebef80dbd7c27e2cb7d4f9214ce40f613941a67e6dcb197a38eea55714a839"
Nov 02 13:24:54.316470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:54.316341    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:54.485847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:54.485803    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:54.739575 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:54.739539    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:56.032327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.032278    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:56.190507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.190469    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:56.262346 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.262316    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"]
Nov 02 13:24:56.263197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.263181    5879 cpu_manager.go:414] "containerMap: removing container" podUID="050c1f0a-55d3-4e78-a9f5-51b0652bc53c" containerName="revoke-keys"
Nov 02 13:24:56.263258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.263200    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050c1f0a-55d3-4e78-a9f5-51b0652bc53c" containerName="revoke-keys"
Nov 02 13:24:56.263439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.263410    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="050c1f0a-55d3-4e78-a9f5-51b0652bc53c" containerName="revoke-keys"
Nov 02 13:24:56.267179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.267160    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"
Nov 02 13:24:56.297615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.297513    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b8a8dc4c-dockercfg-mrvn5\""
Nov 02 13:24:56.297902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.297876    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b8a8dc4c\""
Nov 02 13:24:56.302901 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.302871    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"]
Nov 02 13:24:56.318466 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.317017    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"]
Nov 02 13:24:56.400024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.397827    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg\" (UID: \"038b3a32-6d30-4cd3-9941-8cda31f1b0ea\") " pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"
Nov 02 13:24:56.400024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.397943    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxvr\" (UniqueName: \"kubernetes.io/projected/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-kube-api-access-rpxvr\") pod \"payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg\" (UID: \"038b3a32-6d30-4cd3-9941-8cda31f1b0ea\") " pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"
Nov 02 13:24:56.413014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.410241    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"]
Nov 02 13:24:56.413014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.410380    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"
Nov 02 13:24:56.499823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.499784    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxvr\" (UniqueName: \"kubernetes.io/projected/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-kube-api-access-rpxvr\") pod \"payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg\" (UID: \"038b3a32-6d30-4cd3-9941-8cda31f1b0ea\") " pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"
Nov 02 13:24:56.500082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.499834    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4a3741-0289-4e75-b1bf-2881c27af2ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2\" (UID: \"4b4a3741-0289-4e75-b1bf-2881c27af2ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"
Nov 02 13:24:56.500082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.499977    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndtt\" (UniqueName: \"kubernetes.io/projected/4b4a3741-0289-4e75-b1bf-2881c27af2ce-kube-api-access-tndtt\") pod \"payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2\" (UID: \"4b4a3741-0289-4e75-b1bf-2881c27af2ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"
Nov 02 13:24:56.500238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.500202    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg\" (UID: \"038b3a32-6d30-4cd3-9941-8cda31f1b0ea\") " pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"
Nov 02 13:24:56.501074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.501049    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-plugins-config-volume\") pod \"payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg\" (UID: \"038b3a32-6d30-4cd3-9941-8cda31f1b0ea\") " pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"
Nov 02 13:24:56.505987 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.505962    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:56.525298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.525259    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxvr\" (UniqueName: \"kubernetes.io/projected/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-kube-api-access-rpxvr\") pod \"payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg\" (UID: \"038b3a32-6d30-4cd3-9941-8cda31f1b0ea\") " pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"
Nov 02 13:24:56.584269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.584177    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"
Nov 02 13:24:56.601920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.601884    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4a3741-0289-4e75-b1bf-2881c27af2ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2\" (UID: \"4b4a3741-0289-4e75-b1bf-2881c27af2ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"
Nov 02 13:24:56.602070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.602053    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndtt\" (UniqueName: \"kubernetes.io/projected/4b4a3741-0289-4e75-b1bf-2881c27af2ce-kube-api-access-tndtt\") pod \"payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2\" (UID: \"4b4a3741-0289-4e75-b1bf-2881c27af2ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"
Nov 02 13:24:56.602692 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.602662    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4a3741-0289-4e75-b1bf-2881c27af2ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2\" (UID: \"4b4a3741-0289-4e75-b1bf-2881c27af2ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"
Nov 02 13:24:56.627484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.627453    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndtt\" (UniqueName: \"kubernetes.io/projected/4b4a3741-0289-4e75-b1bf-2881c27af2ce-kube-api-access-tndtt\") pod \"payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2\" (UID: \"4b4a3741-0289-4e75-b1bf-2881c27af2ce\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"
Nov 02 13:24:56.740092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.740053    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"
Nov 02 13:24:56.895458 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:24:56.895416    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038b3a32_6d30_4cd3_9941_8cda31f1b0ea.slice/crio-99c638ff58810fdcfb22250b4f97be8308429ce2805a39bce16931d19629cb14 WatchSource:0}: Error finding container 99c638ff58810fdcfb22250b4f97be8308429ce2805a39bce16931d19629cb14: Status 404 returned error can't find the container with id 99c638ff58810fdcfb22250b4f97be8308429ce2805a39bce16931d19629cb14
Nov 02 13:24:56.897508 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:56.897485    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"]
Nov 02 13:24:57.048414 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:24:57.048384    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4a3741_0289_4e75_b1bf_2881c27af2ce.slice/crio-9ff606956ce22f356b07ddeff9c852a51ca468b7578a5df9156c571be3b034f3 WatchSource:0}: Error finding container 9ff606956ce22f356b07ddeff9c852a51ca468b7578a5df9156c571be3b034f3: Status 404 returned error can't find the container with id 9ff606956ce22f356b07ddeff9c852a51ca468b7578a5df9156c571be3b034f3
Nov 02 13:24:57.050356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:57.050327    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"]
Nov 02 13:24:57.053078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:57.053049    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg" event={"ID":"038b3a32-6d30-4cd3-9941-8cda31f1b0ea","Type":"ContainerStarted","Data":"13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5"}
Nov 02 13:24:57.053189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:57.053118    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg" event={"ID":"038b3a32-6d30-4cd3-9941-8cda31f1b0ea","Type":"ContainerStarted","Data":"99c638ff58810fdcfb22250b4f97be8308429ce2805a39bce16931d19629cb14"}
Nov 02 13:24:57.053321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:57.053263    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"
Nov 02 13:24:57.102757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:57.102712    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg" podStartSLOduration=1.102698824 podStartE2EDuration="1.102698824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:24:57.099005441 +0000 UTC m=+3443.205708282" watchObservedRunningTime="2026-11-02 13:24:57.102698824 +0000 UTC m=+3443.209401675"
Nov 02 13:24:58.065664 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:58.065628    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2" event={"ID":"4b4a3741-0289-4e75-b1bf-2881c27af2ce","Type":"ContainerStarted","Data":"de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc"}
Nov 02 13:24:58.066110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:58.065682    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2" event={"ID":"4b4a3741-0289-4e75-b1bf-2881c27af2ce","Type":"ContainerStarted","Data":"9ff606956ce22f356b07ddeff9c852a51ca468b7578a5df9156c571be3b034f3"}
Nov 02 13:24:58.066110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:58.065734    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"
Nov 02 13:24:58.116023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:58.115922    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2" podStartSLOduration=2.115904186 podStartE2EDuration="2.115904186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:24:58.112798282 +0000 UTC m=+3444.219501130" watchObservedRunningTime="2026-11-02 13:24:58.115904186 +0000 UTC m=+3444.222607036"
Nov 02 13:24:58.224242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:58.224205    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:24:59.935520 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:24:59.935479    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:00.308338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:00.308247    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:01.528446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:01.528414    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:03.065470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:03.065432    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:04.747628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:04.747596    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:06.337506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:06.337474    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:07.268525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:07.268484    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:08.046240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:08.046200    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:08.070961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:08.070930    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"
Nov 02 13:25:09.077329 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:09.077287    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"
Nov 02 13:25:09.925290 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:09.925232    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:11.235420 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:11.235390    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:11.373898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:11.373863    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:11.661874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:11.661842    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:13.541279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:13.541246    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:14.823375 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:25:14.823325    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:25:15.227094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:15.227058    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:16.795201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:16.795162    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:18.555531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:18.555488    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:20.119570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:20.119537    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:21.750406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:21.750362    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:22.023118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:22.023028    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:22.870774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:22.870729    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:23.928598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:23.928555    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:25.936680 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:25.936637    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:27.465313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:27.465273    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:28.971800 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:28.971766    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:29.098355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:29.098322    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:30.803516 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:30.803482    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:32.361547 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:32.361496    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:34.049764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:34.049711    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:35.640813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:35.640781    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:37.293549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:37.293501    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:37.743099 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:37.743065    5879 scope.go:122] "RemoveContainer" containerID="d7e3ebe43ca793219fc6f9c0acd816226fc3d46a40670c9aefa996180a2ec5b9"
Nov 02 13:25:37.754258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:37.754225    5879 scope.go:122] "RemoveContainer" containerID="7dab98508a0f87cd0721b1aa12a4add52cad9dbd80403cefbddddebf5a5cf072"
Nov 02 13:25:38.998341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:38.998267    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:40.600531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:40.600490    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:40.801373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:40.801335    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:41.130265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:41.130230    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:42.255560 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:42.255526    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:43.867595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:43.867554    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:44.823350 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:25:44.823303    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:25:45.555908 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:45.555869    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:46.821479 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:46.821440    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"]
Nov 02 13:25:46.831421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:46.831376    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:46.849112 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:46.849086    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"]
Nov 02 13:25:46.858238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:46.858200    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:25:46.858238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:46.858216    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:25:46.936180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:46.936146    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af934d9e-ww4nl\" (UID: \"095ba194-3832-46fd-8b37-60ea53723c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:46.936525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:46.936503    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af934d9e-ww4nl\" (UID: \"095ba194-3832-46fd-8b37-60ea53723c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:46.936589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:46.936543    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25k4\" (UniqueName: \"kubernetes.io/projected/095ba194-3832-46fd-8b37-60ea53723c32-kube-api-access-s25k4\") pod \"maas-api-revoke-sub-af934d9e-ww4nl\" (UID: \"095ba194-3832-46fd-8b37-60ea53723c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:47.038103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:47.038066    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af934d9e-ww4nl\" (UID: \"095ba194-3832-46fd-8b37-60ea53723c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:47.038103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:47.038107    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s25k4\" (UniqueName: \"kubernetes.io/projected/095ba194-3832-46fd-8b37-60ea53723c32-kube-api-access-s25k4\") pod \"maas-api-revoke-sub-af934d9e-ww4nl\" (UID: \"095ba194-3832-46fd-8b37-60ea53723c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:47.038345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:47.038141    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af934d9e-ww4nl\" (UID: \"095ba194-3832-46fd-8b37-60ea53723c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:47.038868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:47.038845    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af934d9e-ww4nl\" (UID: \"095ba194-3832-46fd-8b37-60ea53723c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:47.040699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:47.040673    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af934d9e-ww4nl\" (UID: \"095ba194-3832-46fd-8b37-60ea53723c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:47.063818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:47.063791    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25k4\" (UniqueName: \"kubernetes.io/projected/095ba194-3832-46fd-8b37-60ea53723c32-kube-api-access-s25k4\") pod \"maas-api-revoke-sub-af934d9e-ww4nl\" (UID: \"095ba194-3832-46fd-8b37-60ea53723c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:47.153273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:47.153242    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:47.186699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:47.186635    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:47.487382 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:25:47.487348    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod095ba194_3832_46fd_8b37_60ea53723c32.slice/crio-6cd8f99f8e72eade1222b0f07cfaad0d5d3d06c3a881e6cce69194b607ea5190 WatchSource:0}: Error finding container 6cd8f99f8e72eade1222b0f07cfaad0d5d3d06c3a881e6cce69194b607ea5190: Status 404 returned error can't find the container with id 6cd8f99f8e72eade1222b0f07cfaad0d5d3d06c3a881e6cce69194b607ea5190
Nov 02 13:25:47.487502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:47.487464    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"]
Nov 02 13:25:47.579534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:47.579494    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl" event={"ID":"095ba194-3832-46fd-8b37-60ea53723c32","Type":"ContainerStarted","Data":"6cd8f99f8e72eade1222b0f07cfaad0d5d3d06c3a881e6cce69194b607ea5190"}
Nov 02 13:25:48.589506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:48.589470    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl" event={"ID":"095ba194-3832-46fd-8b37-60ea53723c32","Type":"ContainerStarted","Data":"f3c736a8ed3b033430b2f84012a71605eaaeb8b58f88f6c6a545b8498eeecf6e"}
Nov 02 13:25:48.635931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:48.635865    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl" podStartSLOduration=2.6358516180000002 podStartE2EDuration="2.635851618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:25:48.633112583 +0000 UTC m=+3494.739815432" watchObservedRunningTime="2026-11-02 13:25:48.635851618 +0000 UTC m=+3494.742554517"
Nov 02 13:25:48.837381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:48.837304    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:49.601303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:49.601260    5879 generic.go:356] "Generic (PLEG): container finished" podID="095ba194-3832-46fd-8b37-60ea53723c32" containerID="f3c736a8ed3b033430b2f84012a71605eaaeb8b58f88f6c6a545b8498eeecf6e" exitCode=0
Nov 02 13:25:49.601741 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:49.601339    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl" event={"ID":"095ba194-3832-46fd-8b37-60ea53723c32","Type":"ContainerDied","Data":"f3c736a8ed3b033430b2f84012a71605eaaeb8b58f88f6c6a545b8498eeecf6e"}
Nov 02 13:25:50.428240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:50.428207    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:50.783280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:50.783252    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:50.897743 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:50.897706    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/095ba194-3832-46fd-8b37-60ea53723c32-kube-api-access-s25k4\" (UniqueName: \"kubernetes.io/projected/095ba194-3832-46fd-8b37-60ea53723c32-kube-api-access-s25k4\") pod \"095ba194-3832-46fd-8b37-60ea53723c32\" (UID: \"095ba194-3832-46fd-8b37-60ea53723c32\") "
Nov 02 13:25:50.897932 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:50.897905    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-openshift-ca\") pod \"095ba194-3832-46fd-8b37-60ea53723c32\" (UID: \"095ba194-3832-46fd-8b37-60ea53723c32\") "
Nov 02 13:25:50.898050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:50.898031    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-xks-ca\") pod \"095ba194-3832-46fd-8b37-60ea53723c32\" (UID: \"095ba194-3832-46fd-8b37-60ea53723c32\") "
Nov 02 13:25:50.898333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:50.898297    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-openshift-ca" pod "095ba194-3832-46fd-8b37-60ea53723c32" (UID: "095ba194-3832-46fd-8b37-60ea53723c32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:25:50.898650 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:50.898625    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:25:50.900190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:50.900166    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095ba194-3832-46fd-8b37-60ea53723c32-kube-api-access-s25k4" pod "095ba194-3832-46fd-8b37-60ea53723c32" (UID: "095ba194-3832-46fd-8b37-60ea53723c32"). InnerVolumeSpecName "kube-api-access-s25k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:25:50.900190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:50.900182    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-xks-ca" pod "095ba194-3832-46fd-8b37-60ea53723c32" (UID: "095ba194-3832-46fd-8b37-60ea53723c32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:25:51.000764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:51.000664    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s25k4\" (UniqueName: \"kubernetes.io/projected/095ba194-3832-46fd-8b37-60ea53723c32-kube-api-access-s25k4\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:25:51.000764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:51.000703    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/095ba194-3832-46fd-8b37-60ea53723c32-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:25:51.618630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:51.618601    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"
Nov 02 13:25:51.618630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:51.618626    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl" event={"ID":"095ba194-3832-46fd-8b37-60ea53723c32","Type":"ContainerDied","Data":"6cd8f99f8e72eade1222b0f07cfaad0d5d3d06c3a881e6cce69194b607ea5190"}
Nov 02 13:25:51.618878 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:51.618672    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cd8f99f8e72eade1222b0f07cfaad0d5d3d06c3a881e6cce69194b607ea5190"
Nov 02 13:25:51.960238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:51.960132    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:53.632676 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:53.632634    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:55.032237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:55.032198    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"]
Nov 02 13:25:55.045802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:55.045771    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19751a1d-pnk6q"]
Nov 02 13:25:55.062248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:55.062218    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"]
Nov 02 13:25:55.062454 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:55.062431    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" podUID="d5a30efd-4594-4665-b620-61748d33306f" containerName="istio-proxy" containerID="cri-o://7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677" gracePeriod=30
Nov 02 13:25:55.214800 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:55.214767    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:56.019099 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:56.019061    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="574d8033-d2ce-4b22-b3c0-a8ac82d989da" path="/var/lib/kubelet/pods/574d8033-d2ce-4b22-b3c0-a8ac82d989da/volumes"
Nov 02 13:25:56.111364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:56.111326    5879 patch_prober.go:29] interesting pod/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:25:56.111858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:56.111392    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" podUID="d5a30efd-4594-4665-b620-61748d33306f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:25:56.861289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:56.861244    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:57.738541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:57.738499    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"]
Nov 02 13:25:57.805037 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:57.804948    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:58.251332 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.251298    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"]
Nov 02 13:25:58.252192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.252174    5879 cpu_manager.go:414] "containerMap: removing container" podUID="095ba194-3832-46fd-8b37-60ea53723c32" containerName="revoke-keys"
Nov 02 13:25:58.252192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.252192    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="095ba194-3832-46fd-8b37-60ea53723c32" containerName="revoke-keys"
Nov 02 13:25:58.252334 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.252315    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="095ba194-3832-46fd-8b37-60ea53723c32" containerName="revoke-keys"
Nov 02 13:25:58.256197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.256177    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:25:58.273100 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.273071    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:25:58.277747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.277717    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"]
Nov 02 13:25:58.286102 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.286067    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:25:58.318041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.317967    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93544e15-db36-48e1-817a-76755f1a355f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb994f5e-r2vr4\" (UID: \"93544e15-db36-48e1-817a-76755f1a355f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:25:58.318253 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.318056    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnzpc\" (UniqueName: \"kubernetes.io/projected/93544e15-db36-48e1-817a-76755f1a355f-kube-api-access-gnzpc\") pod \"maas-api-revoke-sub-cb994f5e-r2vr4\" (UID: \"93544e15-db36-48e1-817a-76755f1a355f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:25:58.318253 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.318226    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93544e15-db36-48e1-817a-76755f1a355f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb994f5e-r2vr4\" (UID: \"93544e15-db36-48e1-817a-76755f1a355f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:25:58.419904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.419870    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93544e15-db36-48e1-817a-76755f1a355f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb994f5e-r2vr4\" (UID: \"93544e15-db36-48e1-817a-76755f1a355f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:25:58.420098 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.419922    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnzpc\" (UniqueName: \"kubernetes.io/projected/93544e15-db36-48e1-817a-76755f1a355f-kube-api-access-gnzpc\") pod \"maas-api-revoke-sub-cb994f5e-r2vr4\" (UID: \"93544e15-db36-48e1-817a-76755f1a355f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:25:58.420098 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.419981    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93544e15-db36-48e1-817a-76755f1a355f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb994f5e-r2vr4\" (UID: \"93544e15-db36-48e1-817a-76755f1a355f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:25:58.420673 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.420648    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93544e15-db36-48e1-817a-76755f1a355f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb994f5e-r2vr4\" (UID: \"93544e15-db36-48e1-817a-76755f1a355f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:25:58.422703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.422678    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93544e15-db36-48e1-817a-76755f1a355f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb994f5e-r2vr4\" (UID: \"93544e15-db36-48e1-817a-76755f1a355f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:25:58.445346 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.445314    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnzpc\" (UniqueName: \"kubernetes.io/projected/93544e15-db36-48e1-817a-76755f1a355f-kube-api-access-gnzpc\") pod \"maas-api-revoke-sub-cb994f5e-r2vr4\" (UID: \"93544e15-db36-48e1-817a-76755f1a355f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:25:58.571581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.571463    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:25:58.716923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.716887    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:25:58.886669 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:25:58.886630    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93544e15_db36_48e1_817a_76755f1a355f.slice/crio-a78bb5ad4b88efe56fb04c366a9560912026c427eec70477b25e9e92041e30dc WatchSource:0}: Error finding container a78bb5ad4b88efe56fb04c366a9560912026c427eec70477b25e9e92041e30dc: Status 404 returned error can't find the container with id a78bb5ad4b88efe56fb04c366a9560912026c427eec70477b25e9e92041e30dc
Nov 02 13:25:58.887442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:58.887320    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"]
Nov 02 13:25:59.701134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:59.701100    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4" event={"ID":"93544e15-db36-48e1-817a-76755f1a355f","Type":"ContainerStarted","Data":"ff309e88a243458857fd45985656538da6737bc71a87e6c52cfffcb1f2e866a3"}
Nov 02 13:25:59.701386 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:59.701165    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4" event={"ID":"93544e15-db36-48e1-817a-76755f1a355f","Type":"ContainerStarted","Data":"a78bb5ad4b88efe56fb04c366a9560912026c427eec70477b25e9e92041e30dc"}
Nov 02 13:25:59.747360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:25:59.747304    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4" podStartSLOduration=1.747289858 podStartE2EDuration="1.747289858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:25:59.744352882 +0000 UTC m=+3505.851055736" watchObservedRunningTime="2026-11-02 13:25:59.747289858 +0000 UTC m=+3505.853992729"
Nov 02 13:26:00.360927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.360900    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 13:26:00.398852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.398814    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:00.456190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.456151    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-certs\") pod \"d5a30efd-4594-4665-b620-61748d33306f\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") "
Nov 02 13:26:00.456413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.456265    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-istio-token\" (UniqueName: \"kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-istio-token\") pod \"d5a30efd-4594-4665-b620-61748d33306f\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") "
Nov 02 13:26:00.456413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.456342    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-credential-socket\") pod \"d5a30efd-4594-4665-b620-61748d33306f\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") "
Nov 02 13:26:00.456413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.456392    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-data\") pod \"d5a30efd-4594-4665-b620-61748d33306f\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") "
Nov 02 13:26:00.456615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.456425    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-socket\") pod \"d5a30efd-4594-4665-b620-61748d33306f\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") "
Nov 02 13:26:00.456615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.456473    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-certs" pod "d5a30efd-4594-4665-b620-61748d33306f" (UID: "d5a30efd-4594-4665-b620-61748d33306f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:00.456615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.456583    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-envoy\") pod \"d5a30efd-4594-4665-b620-61748d33306f\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") "
Nov 02 13:26:00.456838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.456674    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a30efd-4594-4665-b620-61748d33306f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a30efd-4594-4665-b620-61748d33306f-istiod-ca-cert\") pod \"d5a30efd-4594-4665-b620-61748d33306f\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") "
Nov 02 13:26:00.456838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.456725    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-socket" pod "d5a30efd-4594-4665-b620-61748d33306f" (UID: "d5a30efd-4594-4665-b620-61748d33306f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:00.456838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.456745    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-credential-socket" pod "d5a30efd-4594-4665-b620-61748d33306f" (UID: "d5a30efd-4594-4665-b620-61748d33306f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:00.456838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.456755    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-kube-api-access-l7m26\" (UniqueName: \"kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-kube-api-access-l7m26\") pod \"d5a30efd-4594-4665-b620-61748d33306f\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") "
Nov 02 13:26:00.457102 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.456851    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5a30efd-4594-4665-b620-61748d33306f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a30efd-4594-4665-b620-61748d33306f-istio-podinfo\") pod \"d5a30efd-4594-4665-b620-61748d33306f\" (UID: \"d5a30efd-4594-4665-b620-61748d33306f\") "
Nov 02 13:26:00.457168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.457134    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a30efd-4594-4665-b620-61748d33306f-istiod-ca-cert" pod "d5a30efd-4594-4665-b620-61748d33306f" (UID: "d5a30efd-4594-4665-b620-61748d33306f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:26:00.457469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.457441    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-data" pod "d5a30efd-4594-4665-b620-61748d33306f" (UID: "d5a30efd-4594-4665-b620-61748d33306f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:00.457835 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.457810    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:00.457949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.457840    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:00.457949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.457855    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:00.457949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.457869    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:00.457949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.457882    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a30efd-4594-4665-b620-61748d33306f-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:00.459531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.459498    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5a30efd-4594-4665-b620-61748d33306f-istio-podinfo" pod "d5a30efd-4594-4665-b620-61748d33306f" (UID: "d5a30efd-4594-4665-b620-61748d33306f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:26:00.460182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.460160    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-kube-api-access-l7m26" pod "d5a30efd-4594-4665-b620-61748d33306f" (UID: "d5a30efd-4594-4665-b620-61748d33306f"). InnerVolumeSpecName "kube-api-access-l7m26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:00.460470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.460442    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-istio-token" pod "d5a30efd-4594-4665-b620-61748d33306f" (UID: "d5a30efd-4594-4665-b620-61748d33306f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:00.460598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.460496    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-envoy" pod "d5a30efd-4594-4665-b620-61748d33306f" (UID: "d5a30efd-4594-4665-b620-61748d33306f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:00.559669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.559626    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a30efd-4594-4665-b620-61748d33306f-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:00.559669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.559658    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7m26\" (UniqueName: \"kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-kube-api-access-l7m26\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:00.559669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.559671    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a30efd-4594-4665-b620-61748d33306f-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:00.559939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.559684    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a30efd-4594-4665-b620-61748d33306f-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:00.711140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.711094    5879 generic.go:356] "Generic (PLEG): container finished" podID="93544e15-db36-48e1-817a-76755f1a355f" containerID="ff309e88a243458857fd45985656538da6737bc71a87e6c52cfffcb1f2e866a3" exitCode=0
Nov 02 13:26:00.711356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.711171    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4" event={"ID":"93544e15-db36-48e1-817a-76755f1a355f","Type":"ContainerDied","Data":"ff309e88a243458857fd45985656538da6737bc71a87e6c52cfffcb1f2e866a3"}
Nov 02 13:26:00.713013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.712972    5879 generic.go:356] "Generic (PLEG): container finished" podID="d5a30efd-4594-4665-b620-61748d33306f" containerID="7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677" exitCode=0
Nov 02 13:26:00.713156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.713029    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" event={"ID":"d5a30efd-4594-4665-b620-61748d33306f","Type":"ContainerDied","Data":"7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677"}
Nov 02 13:26:00.713156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.713086    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx" event={"ID":"d5a30efd-4594-4665-b620-61748d33306f","Type":"ContainerDied","Data":"32e5a82353867740050cf3b277a05459fbb332999dd4c78f484bae854b994f03"}
Nov 02 13:26:00.713156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.713114    5879 scope.go:122] "RemoveContainer" containerID="7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677"
Nov 02 13:26:00.713156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.713037    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"
Nov 02 13:26:00.727287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.727267    5879 scope.go:122] "RemoveContainer" containerID="7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677"
Nov 02 13:26:00.727588 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:26:00.727559    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677\": container with ID starting with 7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677 not found: ID does not exist" containerID="7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677"
Nov 02 13:26:00.727699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.727591    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677"} err="failed to get container status \"7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677\": rpc error: code = NotFound desc = could not find container \"7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677\": container with ID starting with 7ba70e8e87454a8ee239fac9b206b8cf42a36a3cbb81e7d12afa5a0145fcc677 not found: ID does not exist"
Nov 02 13:26:00.801030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.800981    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"]
Nov 02 13:26:00.817165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:00.817122    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-843978-openshift-default-7965f48c7f-98xtx"]
Nov 02 13:26:01.900472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:01.900443    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:26:01.978919 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:01.978822    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93544e15-db36-48e1-817a-76755f1a355f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93544e15-db36-48e1-817a-76755f1a355f-maas-api-openshift-ca\") pod \"93544e15-db36-48e1-817a-76755f1a355f\" (UID: \"93544e15-db36-48e1-817a-76755f1a355f\") "
Nov 02 13:26:01.979127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:01.978971    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93544e15-db36-48e1-817a-76755f1a355f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93544e15-db36-48e1-817a-76755f1a355f-maas-api-xks-ca\") pod \"93544e15-db36-48e1-817a-76755f1a355f\" (UID: \"93544e15-db36-48e1-817a-76755f1a355f\") "
Nov 02 13:26:01.979127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:01.979095    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93544e15-db36-48e1-817a-76755f1a355f-kube-api-access-gnzpc\" (UniqueName: \"kubernetes.io/projected/93544e15-db36-48e1-817a-76755f1a355f-kube-api-access-gnzpc\") pod \"93544e15-db36-48e1-817a-76755f1a355f\" (UID: \"93544e15-db36-48e1-817a-76755f1a355f\") "
Nov 02 13:26:01.979321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:01.979290    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93544e15-db36-48e1-817a-76755f1a355f-maas-api-openshift-ca" pod "93544e15-db36-48e1-817a-76755f1a355f" (UID: "93544e15-db36-48e1-817a-76755f1a355f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:26:01.979767 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:01.979738    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93544e15-db36-48e1-817a-76755f1a355f-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:01.981790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:01.981758    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93544e15-db36-48e1-817a-76755f1a355f-maas-api-xks-ca" pod "93544e15-db36-48e1-817a-76755f1a355f" (UID: "93544e15-db36-48e1-817a-76755f1a355f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:26:01.981920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:01.981886    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93544e15-db36-48e1-817a-76755f1a355f-kube-api-access-gnzpc" pod "93544e15-db36-48e1-817a-76755f1a355f" (UID: "93544e15-db36-48e1-817a-76755f1a355f"). InnerVolumeSpecName "kube-api-access-gnzpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:02.019737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:02.019699    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a30efd-4594-4665-b620-61748d33306f" path="/var/lib/kubelet/pods/d5a30efd-4594-4665-b620-61748d33306f/volumes"
Nov 02 13:26:02.081214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:02.081174    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93544e15-db36-48e1-817a-76755f1a355f-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:02.081214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:02.081214    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnzpc\" (UniqueName: \"kubernetes.io/projected/93544e15-db36-48e1-817a-76755f1a355f-kube-api-access-gnzpc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:02.151868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:02.151835    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:02.731248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:02.731211    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"
Nov 02 13:26:02.731432 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:02.731224    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4" event={"ID":"93544e15-db36-48e1-817a-76755f1a355f","Type":"ContainerDied","Data":"a78bb5ad4b88efe56fb04c366a9560912026c427eec70477b25e9e92041e30dc"}
Nov 02 13:26:02.731432 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:02.731340    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a78bb5ad4b88efe56fb04c366a9560912026c427eec70477b25e9e92041e30dc"
Nov 02 13:26:03.879952 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:03.879902    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:05.685029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:05.684967    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:07.560282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.560238    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:07.579518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.579489    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"]
Nov 02 13:26:07.580370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.580352    5879 cpu_manager.go:414] "containerMap: removing container" podUID="d5a30efd-4594-4665-b620-61748d33306f" containerName="istio-proxy"
Nov 02 13:26:07.580438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.580372    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a30efd-4594-4665-b620-61748d33306f" containerName="istio-proxy"
Nov 02 13:26:07.580438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.580423    5879 cpu_manager.go:414] "containerMap: removing container" podUID="93544e15-db36-48e1-817a-76755f1a355f" containerName="revoke-keys"
Nov 02 13:26:07.580438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.580428    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93544e15-db36-48e1-817a-76755f1a355f" containerName="revoke-keys"
Nov 02 13:26:07.580584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.580573    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="93544e15-db36-48e1-817a-76755f1a355f" containerName="revoke-keys"
Nov 02 13:26:07.580624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.580593    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a30efd-4594-4665-b620-61748d33306f" containerName="istio-proxy"
Nov 02 13:26:07.584421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.584404    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:07.605441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.605405    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"]
Nov 02 13:26:07.615038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.614827    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:26:07.615038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.614836    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:26:07.670293 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.670247    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-513d69a5-xrwb6\" (UID: \"9435ef83-7afb-488c-9a2e-93aef954568b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:07.670649 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.670616    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-513d69a5-xrwb6\" (UID: \"9435ef83-7afb-488c-9a2e-93aef954568b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:07.670716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.670683    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v87wt\" (UniqueName: \"kubernetes.io/projected/9435ef83-7afb-488c-9a2e-93aef954568b-kube-api-access-v87wt\") pod \"maas-api-revoke-sub-513d69a5-xrwb6\" (UID: \"9435ef83-7afb-488c-9a2e-93aef954568b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:07.772615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.772580    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-513d69a5-xrwb6\" (UID: \"9435ef83-7afb-488c-9a2e-93aef954568b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:07.772834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.772637    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v87wt\" (UniqueName: \"kubernetes.io/projected/9435ef83-7afb-488c-9a2e-93aef954568b-kube-api-access-v87wt\") pod \"maas-api-revoke-sub-513d69a5-xrwb6\" (UID: \"9435ef83-7afb-488c-9a2e-93aef954568b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:07.772834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.772756    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-513d69a5-xrwb6\" (UID: \"9435ef83-7afb-488c-9a2e-93aef954568b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:07.773300 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.773281    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-513d69a5-xrwb6\" (UID: \"9435ef83-7afb-488c-9a2e-93aef954568b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:07.775315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.775297    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-513d69a5-xrwb6\" (UID: \"9435ef83-7afb-488c-9a2e-93aef954568b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:07.797662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.797631    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v87wt\" (UniqueName: \"kubernetes.io/projected/9435ef83-7afb-488c-9a2e-93aef954568b-kube-api-access-v87wt\") pod \"maas-api-revoke-sub-513d69a5-xrwb6\" (UID: \"9435ef83-7afb-488c-9a2e-93aef954568b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:07.899741 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:07.899705    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:08.215439 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:26:08.215409    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9435ef83_7afb_488c_9a2e_93aef954568b.slice/crio-e75473c2b49bb126fb52c425cdfbce0fa7370537cfd76f971496c94f6ece1cc2 WatchSource:0}: Error finding container e75473c2b49bb126fb52c425cdfbce0fa7370537cfd76f971496c94f6ece1cc2: Status 404 returned error can't find the container with id e75473c2b49bb126fb52c425cdfbce0fa7370537cfd76f971496c94f6ece1cc2
Nov 02 13:26:08.216942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:08.216915    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"]
Nov 02 13:26:08.781469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:08.781432    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6" event={"ID":"9435ef83-7afb-488c-9a2e-93aef954568b","Type":"ContainerStarted","Data":"de82f29d198ed62e356b58de68539ebe3a88de14c9412adc404c62992d199101"}
Nov 02 13:26:08.782161 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:08.781488    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6" event={"ID":"9435ef83-7afb-488c-9a2e-93aef954568b","Type":"ContainerStarted","Data":"e75473c2b49bb126fb52c425cdfbce0fa7370537cfd76f971496c94f6ece1cc2"}
Nov 02 13:26:08.827115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:08.827060    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6" podStartSLOduration=1.82704731 podStartE2EDuration="1.82704731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:26:08.825329199 +0000 UTC m=+3514.932032032" watchObservedRunningTime="2026-11-02 13:26:08.82704731 +0000 UTC m=+3514.933750160"
Nov 02 13:26:09.315444 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:09.315409    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:09.791648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:09.791612    5879 generic.go:356] "Generic (PLEG): container finished" podID="9435ef83-7afb-488c-9a2e-93aef954568b" containerID="de82f29d198ed62e356b58de68539ebe3a88de14c9412adc404c62992d199101" exitCode=0
Nov 02 13:26:09.792147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:09.791701    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6" event={"ID":"9435ef83-7afb-488c-9a2e-93aef954568b","Type":"ContainerDied","Data":"de82f29d198ed62e356b58de68539ebe3a88de14c9412adc404c62992d199101"}
Nov 02 13:26:10.969812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:10.969783    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:11.108469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.108381    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:11.125132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.125089    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-openshift-ca\") pod \"9435ef83-7afb-488c-9a2e-93aef954568b\" (UID: \"9435ef83-7afb-488c-9a2e-93aef954568b\") "
Nov 02 13:26:11.125445 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.125403    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9435ef83-7afb-488c-9a2e-93aef954568b-kube-api-access-v87wt\" (UniqueName: \"kubernetes.io/projected/9435ef83-7afb-488c-9a2e-93aef954568b-kube-api-access-v87wt\") pod \"9435ef83-7afb-488c-9a2e-93aef954568b\" (UID: \"9435ef83-7afb-488c-9a2e-93aef954568b\") "
Nov 02 13:26:11.125595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.125563    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-xks-ca\") pod \"9435ef83-7afb-488c-9a2e-93aef954568b\" (UID: \"9435ef83-7afb-488c-9a2e-93aef954568b\") "
Nov 02 13:26:11.125673 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.125606    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-openshift-ca" pod "9435ef83-7afb-488c-9a2e-93aef954568b" (UID: "9435ef83-7afb-488c-9a2e-93aef954568b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:26:11.126361 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.126335    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:11.128029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.127963    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9435ef83-7afb-488c-9a2e-93aef954568b-kube-api-access-v87wt" pod "9435ef83-7afb-488c-9a2e-93aef954568b" (UID: "9435ef83-7afb-488c-9a2e-93aef954568b"). InnerVolumeSpecName "kube-api-access-v87wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:11.128137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.128118    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-xks-ca" pod "9435ef83-7afb-488c-9a2e-93aef954568b" (UID: "9435ef83-7afb-488c-9a2e-93aef954568b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:26:11.227807 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.227774    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v87wt\" (UniqueName: \"kubernetes.io/projected/9435ef83-7afb-488c-9a2e-93aef954568b-kube-api-access-v87wt\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:11.227807 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.227805    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9435ef83-7afb-488c-9a2e-93aef954568b-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:11.808096 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.808061    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"
Nov 02 13:26:11.808306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.808058    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6" event={"ID":"9435ef83-7afb-488c-9a2e-93aef954568b","Type":"ContainerDied","Data":"e75473c2b49bb126fb52c425cdfbce0fa7370537cfd76f971496c94f6ece1cc2"}
Nov 02 13:26:11.808306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:11.808220    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e75473c2b49bb126fb52c425cdfbce0fa7370537cfd76f971496c94f6ece1cc2"
Nov 02 13:26:12.770682 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:12.770637    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk" podUID="dfcead4b-186c-4c78-8d1d-93d07d3c450a" containerName="main" containerID="cri-o://ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313" gracePeriod=45
Nov 02 13:26:12.872165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:12.872128    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:13.088678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.088651    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:26:13.155104 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.155066    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tmp-dir\") pod \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") "
Nov 02 13:26:13.155396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.155184    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfcead4b-186c-4c78-8d1d-93d07d3c450a-kube-api-access-745md\" (UniqueName: \"kubernetes.io/projected/dfcead4b-186c-4c78-8d1d-93d07d3c450a-kube-api-access-745md\") pod \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") "
Nov 02 13:26:13.155396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.155249    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-model-cache\") pod \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") "
Nov 02 13:26:13.155396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.155293    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tls-certs\" (UniqueName: \"kubernetes.io/secret/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tls-certs\") pod \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") "
Nov 02 13:26:13.155612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.155424    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tmp-dir" pod "dfcead4b-186c-4c78-8d1d-93d07d3c450a" (UID: "dfcead4b-186c-4c78-8d1d-93d07d3c450a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:13.155612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.155440    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-dshm\") pod \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") "
Nov 02 13:26:13.155888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.155865    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-model-cache" pod "dfcead4b-186c-4c78-8d1d-93d07d3c450a" (UID: "dfcead4b-186c-4c78-8d1d-93d07d3c450a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:13.156348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.156323    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:13.156465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.156354    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:13.158523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.158492    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-dshm" pod "dfcead4b-186c-4c78-8d1d-93d07d3c450a" (UID: "dfcead4b-186c-4c78-8d1d-93d07d3c450a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:13.158523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.158511    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfcead4b-186c-4c78-8d1d-93d07d3c450a-kube-api-access-745md" pod "dfcead4b-186c-4c78-8d1d-93d07d3c450a" (UID: "dfcead4b-186c-4c78-8d1d-93d07d3c450a"). InnerVolumeSpecName "kube-api-access-745md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:13.158720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.158677    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tls-certs" pod "dfcead4b-186c-4c78-8d1d-93d07d3c450a" (UID: "dfcead4b-186c-4c78-8d1d-93d07d3c450a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:26:13.256897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.256864    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-home\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-home\") pod \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\" (UID: \"dfcead4b-186c-4c78-8d1d-93d07d3c450a\") "
Nov 02 13:26:13.257184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.257162    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-home" pod "dfcead4b-186c-4c78-8d1d-93d07d3c450a" (UID: "dfcead4b-186c-4c78-8d1d-93d07d3c450a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:13.257490 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.257474    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfcead4b-186c-4c78-8d1d-93d07d3c450a-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:13.257530 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.257493    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:13.257530 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.257501    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfcead4b-186c-4c78-8d1d-93d07d3c450a-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:13.257530 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.257510    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-745md\" (UniqueName: \"kubernetes.io/projected/dfcead4b-186c-4c78-8d1d-93d07d3c450a-kube-api-access-745md\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:13.825581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.825492    5879 generic.go:356] "Generic (PLEG): container finished" podID="dfcead4b-186c-4c78-8d1d-93d07d3c450a" containerID="ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313" exitCode=0
Nov 02 13:26:13.825581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.825534    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"
Nov 02 13:26:13.826075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.825585    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk" event={"ID":"dfcead4b-186c-4c78-8d1d-93d07d3c450a","Type":"ContainerDied","Data":"ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313"}
Nov 02 13:26:13.826075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.825639    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk" event={"ID":"dfcead4b-186c-4c78-8d1d-93d07d3c450a","Type":"ContainerDied","Data":"e4b8c44f0df0eb3f36cc191ccba5eec2c5e3ce3c21728ce97efa77d1f8140047"}
Nov 02 13:26:13.826075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.825665    5879 scope.go:122] "RemoveContainer" containerID="ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313"
Nov 02 13:26:13.838109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.838080    5879 scope.go:122] "RemoveContainer" containerID="ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313"
Nov 02 13:26:13.838418 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:26:13.838396    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313\": container with ID starting with ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313 not found: ID does not exist" containerID="ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313"
Nov 02 13:26:13.838511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.838426    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313"} err="failed to get container status \"ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313\": rpc error: code = NotFound desc = could not find container \"ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313\": container with ID starting with ef18e5f41d7fb5bd2dee27ae7a183e7f39a97c8d18b6cb0db7eb8253ddef5313 not found: ID does not exist"
Nov 02 13:26:13.883615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.883577    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"]
Nov 02 13:26:13.897418 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:13.897387    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e8a402/test-model-e8a402-kserve-85d666df87-sz2bk"]
Nov 02 13:26:14.019384 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:14.019346    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfcead4b-186c-4c78-8d1d-93d07d3c450a" path="/var/lib/kubelet/pods/dfcead4b-186c-4c78-8d1d-93d07d3c450a/volumes"
Nov 02 13:26:14.625779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:14.625735    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:14.822758 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:26:14.822715    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:26:16.388395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:16.388359    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:17.705237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:17.705199    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"]
Nov 02 13:26:17.760681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:17.760645    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:18.267879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:18.267844    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:18.391817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:18.391778    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:19.733249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:19.733221    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:20.009947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:20.009820    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:21.687117 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:21.687078    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:23.035427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:23.035391    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"]
Nov 02 13:26:23.049022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:23.048979    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e49734de-tjmdx"]
Nov 02 13:26:23.520667 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:23.520621    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:24.019330 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:24.019293    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f2c3b6-4105-48d2-a8ae-33df7141fc13" path="/var/lib/kubelet/pods/21f2c3b6-4105-48d2-a8ae-33df7141fc13/volumes"
Nov 02 13:26:24.043612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:24.043580    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"]
Nov 02 13:26:24.059347 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:24.059318    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"]
Nov 02 13:26:24.218689 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:24.218660    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:24.392038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:24.391970    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:25.495869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:25.495818    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:25.776768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:25.776672    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:26.034370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:26.034290    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"]
Nov 02 13:26:26.046916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:26.046887    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d62db4d8-thcb6"]
Nov 02 13:26:27.302544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:27.302492    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:28.019223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:28.019187    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="142cb7a8-c0b0-40e6-97f9-5071c87ad3fd" path="/var/lib/kubelet/pods/142cb7a8-c0b0-40e6-97f9-5071c87ad3fd/volumes"
Nov 02 13:26:29.044184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.044138    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9" podUID="f6b54b7e-a38e-4899-8986-4b9d27c427a3" containerName="payload-processing" containerID="cri-o://da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8" gracePeriod=25
Nov 02 13:26:29.059616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.059583    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k" podUID="c5749f06-f474-4e93-b640-8eaeadc2ec35" containerName="payload-pre-processing" containerID="cri-o://3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d" gracePeriod=25
Nov 02 13:26:29.299736 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.299645    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"]
Nov 02 13:26:29.370559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.370531    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"
Nov 02 13:26:29.514414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.514380    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b54b7e-a38e-4899-8986-4b9d27c427a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b54b7e-a38e-4899-8986-4b9d27c427a3-plugins-config-volume\") pod \"f6b54b7e-a38e-4899-8986-4b9d27c427a3\" (UID: \"f6b54b7e-a38e-4899-8986-4b9d27c427a3\") "
Nov 02 13:26:29.514608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.514473    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b54b7e-a38e-4899-8986-4b9d27c427a3-kube-api-access-fm5g7\" (UniqueName: \"kubernetes.io/projected/f6b54b7e-a38e-4899-8986-4b9d27c427a3-kube-api-access-fm5g7\") pod \"f6b54b7e-a38e-4899-8986-4b9d27c427a3\" (UID: \"f6b54b7e-a38e-4899-8986-4b9d27c427a3\") "
Nov 02 13:26:29.514884 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.514859    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b54b7e-a38e-4899-8986-4b9d27c427a3-plugins-config-volume" pod "f6b54b7e-a38e-4899-8986-4b9d27c427a3" (UID: "f6b54b7e-a38e-4899-8986-4b9d27c427a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:26:29.515176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.515157    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b54b7e-a38e-4899-8986-4b9d27c427a3-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:29.517523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.517486    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b54b7e-a38e-4899-8986-4b9d27c427a3-kube-api-access-fm5g7" pod "f6b54b7e-a38e-4899-8986-4b9d27c427a3" (UID: "f6b54b7e-a38e-4899-8986-4b9d27c427a3"). InnerVolumeSpecName "kube-api-access-fm5g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:29.616203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.616179    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm5g7\" (UniqueName: \"kubernetes.io/projected/f6b54b7e-a38e-4899-8986-4b9d27c427a3-kube-api-access-fm5g7\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:29.794494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.794461    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"]
Nov 02 13:26:29.795393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.795373    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f6b54b7e-a38e-4899-8986-4b9d27c427a3" containerName="payload-processing"
Nov 02 13:26:29.795393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.795394    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b54b7e-a38e-4899-8986-4b9d27c427a3" containerName="payload-processing"
Nov 02 13:26:29.795543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.795413    5879 cpu_manager.go:414] "containerMap: removing container" podUID="dfcead4b-186c-4c78-8d1d-93d07d3c450a" containerName="main"
Nov 02 13:26:29.795543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.795418    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfcead4b-186c-4c78-8d1d-93d07d3c450a" containerName="main"
Nov 02 13:26:29.795543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.795437    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9435ef83-7afb-488c-9a2e-93aef954568b" containerName="revoke-keys"
Nov 02 13:26:29.795543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.795443    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9435ef83-7afb-488c-9a2e-93aef954568b" containerName="revoke-keys"
Nov 02 13:26:29.795677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.795563    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9435ef83-7afb-488c-9a2e-93aef954568b" containerName="revoke-keys"
Nov 02 13:26:29.795677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.795580    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b54b7e-a38e-4899-8986-4b9d27c427a3" containerName="payload-processing"
Nov 02 13:26:29.795677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.795589    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfcead4b-186c-4c78-8d1d-93d07d3c450a" containerName="main"
Nov 02 13:26:29.799438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.799421    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:29.816449 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.816415    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:26:29.817101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.816765    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:26:29.819079 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.819047    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"]
Nov 02 13:26:29.923259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.923152    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90c87594-tpcqm\" (UID: \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:29.923451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.923389    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmzdz\" (UniqueName: \"kubernetes.io/projected/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-kube-api-access-qmzdz\") pod \"maas-api-revoke-sub-90c87594-tpcqm\" (UID: \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:29.923519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.923495    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90c87594-tpcqm\" (UID: \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:29.970941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.970906    5879 generic.go:356] "Generic (PLEG): container finished" podID="f6b54b7e-a38e-4899-8986-4b9d27c427a3" containerID="da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8" exitCode=0
Nov 02 13:26:29.971144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.970964    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"
Nov 02 13:26:29.971144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.970987    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9" event={"ID":"f6b54b7e-a38e-4899-8986-4b9d27c427a3","Type":"ContainerDied","Data":"da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8"}
Nov 02 13:26:29.971144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.971096    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9" event={"ID":"f6b54b7e-a38e-4899-8986-4b9d27c427a3","Type":"ContainerDied","Data":"0abb5576596c1eda6541259809fc5d88e2c3e5179d8b15c4a062d22ea5f3df89"}
Nov 02 13:26:29.971144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.971128    5879 scope.go:122] "RemoveContainer" containerID="da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8"
Nov 02 13:26:29.984878 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.984856    5879 scope.go:122] "RemoveContainer" containerID="da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8"
Nov 02 13:26:29.985197 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:26:29.985175    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8\": container with ID starting with da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8 not found: ID does not exist" containerID="da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8"
Nov 02 13:26:29.985314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:29.985204    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8"} err="failed to get container status \"da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8\": rpc error: code = NotFound desc = could not find container \"da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8\": container with ID starting with da9e529274bb26da353b52a1f6ef7440deec7c7f85a3ffcc8f4b8adf746136a8 not found: ID does not exist"
Nov 02 13:26:30.025259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.025224    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzdz\" (UniqueName: \"kubernetes.io/projected/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-kube-api-access-qmzdz\") pod \"maas-api-revoke-sub-90c87594-tpcqm\" (UID: \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:30.025441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.025310    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90c87594-tpcqm\" (UID: \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:30.025441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.025432    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90c87594-tpcqm\" (UID: \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:30.026399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.026376    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90c87594-tpcqm\" (UID: \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:30.028439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.028411    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90c87594-tpcqm\" (UID: \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:30.032065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.032037    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"]
Nov 02 13:26:30.034950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.034927    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e8a402-bd7d6fcbf-2tkh9"]
Nov 02 13:26:30.051184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.051162    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmzdz\" (UniqueName: \"kubernetes.io/projected/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-kube-api-access-qmzdz\") pod \"maas-api-revoke-sub-90c87594-tpcqm\" (UID: \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:30.114154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.114131    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:30.119295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.119273    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:30.424899 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:26:30.424838    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2e31b0_7ac4_45a3_a927_0670eea8e6df.slice/crio-9518789266d578bf64987ff8c094755d25a5cd54ac1c2c3691c3f507d47891f2 WatchSource:0}: Error finding container 9518789266d578bf64987ff8c094755d25a5cd54ac1c2c3691c3f507d47891f2: Status 404 returned error can't find the container with id 9518789266d578bf64987ff8c094755d25a5cd54ac1c2c3691c3f507d47891f2
Nov 02 13:26:30.425931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.425910    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"]
Nov 02 13:26:30.981677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.981644    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm" event={"ID":"5e2e31b0-7ac4-45a3-a927-0670eea8e6df","Type":"ContainerStarted","Data":"f620d71a1c1e10c3342cd3b08397f42bab73766aeb08c31b7f3f343f72f482fb"}
Nov 02 13:26:30.981861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:30.981701    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm" event={"ID":"5e2e31b0-7ac4-45a3-a927-0670eea8e6df","Type":"ContainerStarted","Data":"9518789266d578bf64987ff8c094755d25a5cd54ac1c2c3691c3f507d47891f2"}
Nov 02 13:26:31.000343 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.000312    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:31.022037 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.019776    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"]
Nov 02 13:26:31.026119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.026082    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:31.029240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.029190    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm" podStartSLOduration=2.029175362 podStartE2EDuration="2.029175362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:26:31.027695881 +0000 UTC m=+3537.134398734" watchObservedRunningTime="2026-11-02 13:26:31.029175362 +0000 UTC m=+3537.135878230"
Nov 02 13:26:31.039326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.039293    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c1d5885-9r48g\" (UID: \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:31.039845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.039820    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c1d5885-9r48g\" (UID: \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:31.040049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.039980    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d8bp\" (UniqueName: \"kubernetes.io/projected/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-kube-api-access-9d8bp\") pod \"maas-api-revoke-sub-1c1d5885-9r48g\" (UID: \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:31.047569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.047545    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"]
Nov 02 13:26:31.141458 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.141421    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c1d5885-9r48g\" (UID: \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:31.141913 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.141465    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d8bp\" (UniqueName: \"kubernetes.io/projected/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-kube-api-access-9d8bp\") pod \"maas-api-revoke-sub-1c1d5885-9r48g\" (UID: \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:31.141913 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.141609    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c1d5885-9r48g\" (UID: \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:31.142309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.142290    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c1d5885-9r48g\" (UID: \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:31.144050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.144027    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c1d5885-9r48g\" (UID: \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:31.166910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.166877    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d8bp\" (UniqueName: \"kubernetes.io/projected/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-kube-api-access-9d8bp\") pod \"maas-api-revoke-sub-1c1d5885-9r48g\" (UID: \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:31.346351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.346321    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:31.653528 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:26:31.653497    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf1b54a_da6e_4e8e_85d5_4ddc4db02eae.slice/crio-7c8e41df2d25ca2039523c68a4180c5bc3e37379380624c9eb5ae24de1e1bbb2 WatchSource:0}: Error finding container 7c8e41df2d25ca2039523c68a4180c5bc3e37379380624c9eb5ae24de1e1bbb2: Status 404 returned error can't find the container with id 7c8e41df2d25ca2039523c68a4180c5bc3e37379380624c9eb5ae24de1e1bbb2
Nov 02 13:26:31.654561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.654536    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"]
Nov 02 13:26:31.995093 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.995060    5879 generic.go:356] "Generic (PLEG): container finished" podID="5e2e31b0-7ac4-45a3-a927-0670eea8e6df" containerID="f620d71a1c1e10c3342cd3b08397f42bab73766aeb08c31b7f3f343f72f482fb" exitCode=0
Nov 02 13:26:31.995227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.995183    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm" event={"ID":"5e2e31b0-7ac4-45a3-a927-0670eea8e6df","Type":"ContainerDied","Data":"f620d71a1c1e10c3342cd3b08397f42bab73766aeb08c31b7f3f343f72f482fb"}
Nov 02 13:26:31.996691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:31.996667    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g" event={"ID":"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae","Type":"ContainerStarted","Data":"7c8e41df2d25ca2039523c68a4180c5bc3e37379380624c9eb5ae24de1e1bbb2"}
Nov 02 13:26:32.019884 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:32.019852    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b54b7e-a38e-4899-8986-4b9d27c427a3" path="/var/lib/kubelet/pods/f6b54b7e-a38e-4899-8986-4b9d27c427a3/volumes"
Nov 02 13:26:32.104908 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:32.104877    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:32.733678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:32.733637    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq" podUID="4aa69de3-1d00-44f8-af1e-796146e70910" containerName="main" containerID="cri-o://f80bd6e5ff469447517e76f814de3bb949db5b95d99f30885a7742930053aa04" gracePeriod=45
Nov 02 13:26:32.910624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:32.910578    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq" podUID="4aa69de3-1d00-44f8-af1e-796146e70910" containerName="main" probeResult="failure" output="Get \"https://10.134.1.89:8000/ready\": dial tcp 10.134.1.89:8000: connect: connection refused"
Nov 02 13:26:32.992817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:32.992724    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:33.011576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.011537    5879 generic.go:356] "Generic (PLEG): container finished" podID="2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae" containerID="88612ea11f1638b6a4777831ac5b4fe989657f9af5c0566099e5c6ba7871df76" exitCode=0
Nov 02 13:26:33.011734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.011662    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g" event={"ID":"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae","Type":"ContainerDied","Data":"88612ea11f1638b6a4777831ac5b4fe989657f9af5c0566099e5c6ba7871df76"}
Nov 02 13:26:33.013128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.013107    5879 generic.go:356] "Generic (PLEG): container finished" podID="4aa69de3-1d00-44f8-af1e-796146e70910" containerID="f80bd6e5ff469447517e76f814de3bb949db5b95d99f30885a7742930053aa04" exitCode=0
Nov 02 13:26:33.013243 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.013175    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq" event={"ID":"4aa69de3-1d00-44f8-af1e-796146e70910","Type":"ContainerDied","Data":"f80bd6e5ff469447517e76f814de3bb949db5b95d99f30885a7742930053aa04"}
Nov 02 13:26:33.013243 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.013206    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq" event={"ID":"4aa69de3-1d00-44f8-af1e-796146e70910","Type":"ContainerDied","Data":"de8d947b57d76c3c21840c240c523e7fd950b18de17efe3c50b6dd45b3f8ea26"}
Nov 02 13:26:33.013243 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.013229    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de8d947b57d76c3c21840c240c523e7fd950b18de17efe3c50b6dd45b3f8ea26"
Nov 02 13:26:33.033403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.033376    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:26:33.070785 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.070749    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa69de3-1d00-44f8-af1e-796146e70910-kube-api-access-lpjnq\" (UniqueName: \"kubernetes.io/projected/4aa69de3-1d00-44f8-af1e-796146e70910-kube-api-access-lpjnq\") pod \"4aa69de3-1d00-44f8-af1e-796146e70910\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") "
Nov 02 13:26:33.070982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.070799    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-tmp-dir\") pod \"4aa69de3-1d00-44f8-af1e-796146e70910\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") "
Nov 02 13:26:33.070982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.070918    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-home\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-home\") pod \"4aa69de3-1d00-44f8-af1e-796146e70910\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") "
Nov 02 13:26:33.071257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.071227    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-home" pod "4aa69de3-1d00-44f8-af1e-796146e70910" (UID: "4aa69de3-1d00-44f8-af1e-796146e70910"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:33.071257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.071251    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-model-cache\") pod \"4aa69de3-1d00-44f8-af1e-796146e70910\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") "
Nov 02 13:26:33.071462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.071321    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-dshm\") pod \"4aa69de3-1d00-44f8-af1e-796146e70910\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") "
Nov 02 13:26:33.071462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.071389    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa69de3-1d00-44f8-af1e-796146e70910-tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa69de3-1d00-44f8-af1e-796146e70910-tls-certs\") pod \"4aa69de3-1d00-44f8-af1e-796146e70910\" (UID: \"4aa69de3-1d00-44f8-af1e-796146e70910\") "
Nov 02 13:26:33.071597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.071383    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-tmp-dir" pod "4aa69de3-1d00-44f8-af1e-796146e70910" (UID: "4aa69de3-1d00-44f8-af1e-796146e70910"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:33.071597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.071448    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-model-cache" pod "4aa69de3-1d00-44f8-af1e-796146e70910" (UID: "4aa69de3-1d00-44f8-af1e-796146e70910"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:33.072222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.072180    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:33.072382 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.072225    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:33.072382 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.072240    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:33.074328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.074301    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-dshm" pod "4aa69de3-1d00-44f8-af1e-796146e70910" (UID: "4aa69de3-1d00-44f8-af1e-796146e70910"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:33.074431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.074352    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa69de3-1d00-44f8-af1e-796146e70910-kube-api-access-lpjnq" pod "4aa69de3-1d00-44f8-af1e-796146e70910" (UID: "4aa69de3-1d00-44f8-af1e-796146e70910"). InnerVolumeSpecName "kube-api-access-lpjnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:33.074431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.074373    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa69de3-1d00-44f8-af1e-796146e70910-tls-certs" pod "4aa69de3-1d00-44f8-af1e-796146e70910" (UID: "4aa69de3-1d00-44f8-af1e-796146e70910"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:26:33.174420 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.174388    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4aa69de3-1d00-44f8-af1e-796146e70910-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:33.174562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.174423    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4aa69de3-1d00-44f8-af1e-796146e70910-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:33.174562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.174440    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpjnq\" (UniqueName: \"kubernetes.io/projected/4aa69de3-1d00-44f8-af1e-796146e70910-kube-api-access-lpjnq\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:33.186656 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.186625    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:33.275578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.275484    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-kube-api-access-qmzdz\" (UniqueName: \"kubernetes.io/projected/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-kube-api-access-qmzdz\") pod \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\" (UID: \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\") "
Nov 02 13:26:33.275734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.275637    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-xks-ca\") pod \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\" (UID: \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\") "
Nov 02 13:26:33.275784 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.275736    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-openshift-ca\") pod \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\" (UID: \"5e2e31b0-7ac4-45a3-a927-0670eea8e6df\") "
Nov 02 13:26:33.276155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.276128    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-openshift-ca" pod "5e2e31b0-7ac4-45a3-a927-0670eea8e6df" (UID: "5e2e31b0-7ac4-45a3-a927-0670eea8e6df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:26:33.278511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.278486    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-xks-ca" pod "5e2e31b0-7ac4-45a3-a927-0670eea8e6df" (UID: "5e2e31b0-7ac4-45a3-a927-0670eea8e6df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:26:33.278600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.278529    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-kube-api-access-qmzdz" pod "5e2e31b0-7ac4-45a3-a927-0670eea8e6df" (UID: "5e2e31b0-7ac4-45a3-a927-0670eea8e6df"). InnerVolumeSpecName "kube-api-access-qmzdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:33.377949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.377916    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:33.377949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.377945    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmzdz\" (UniqueName: \"kubernetes.io/projected/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-kube-api-access-qmzdz\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:33.377949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:33.377955    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2e31b0-7ac4-45a3-a927-0670eea8e6df-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:34.036580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.036550    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"
Nov 02 13:26:34.037378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.037356    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"
Nov 02 13:26:34.037484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.037388    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm" event={"ID":"5e2e31b0-7ac4-45a3-a927-0670eea8e6df","Type":"ContainerDied","Data":"9518789266d578bf64987ff8c094755d25a5cd54ac1c2c3691c3f507d47891f2"}
Nov 02 13:26:34.037484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.037477    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9518789266d578bf64987ff8c094755d25a5cd54ac1c2c3691c3f507d47891f2"
Nov 02 13:26:34.096936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.096897    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"]
Nov 02 13:26:34.102776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.102752    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-357b30/test-model-357b30-kserve-69d478754c-5g5kq"]
Nov 02 13:26:34.213420 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.213385    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:34.289867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.289785    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-xks-ca\") pod \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\" (UID: \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\") "
Nov 02 13:26:34.290085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.289874    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-openshift-ca\") pod \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\" (UID: \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\") "
Nov 02 13:26:34.290085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.289921    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-kube-api-access-9d8bp\" (UniqueName: \"kubernetes.io/projected/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-kube-api-access-9d8bp\") pod \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\" (UID: \"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae\") "
Nov 02 13:26:34.290274 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.290251    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-openshift-ca" pod "2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae" (UID: "2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:26:34.290634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.290614    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:34.292945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.292918    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-kube-api-access-9d8bp" pod "2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae" (UID: "2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae"). InnerVolumeSpecName "kube-api-access-9d8bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:34.293126 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.292929    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-xks-ca" pod "2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae" (UID: "2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:26:34.395053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.392418    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:34.395053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.392453    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d8bp\" (UniqueName: \"kubernetes.io/projected/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae-kube-api-access-9d8bp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:34.433452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.433415    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:34.649485 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:34.649443    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k" podUID="c5749f06-f474-4e93-b640-8eaeadc2ec35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:26:35.046181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:35.046099    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"
Nov 02 13:26:35.046181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:35.046138    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g" event={"ID":"2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae","Type":"ContainerDied","Data":"7c8e41df2d25ca2039523c68a4180c5bc3e37379380624c9eb5ae24de1e1bbb2"}
Nov 02 13:26:35.046655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:35.046185    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c8e41df2d25ca2039523c68a4180c5bc3e37379380624c9eb5ae24de1e1bbb2"
Nov 02 13:26:35.183579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:35.183543    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:36.018970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:36.018941    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa69de3-1d00-44f8-af1e-796146e70910" path="/var/lib/kubelet/pods/4aa69de3-1d00-44f8-af1e-796146e70910/volumes"
Nov 02 13:26:36.535693 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:36.535648    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:36.949414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:36.949381    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:37.418463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:37.418430    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:37.581634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:37.581592    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:38.104153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:38.104070    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:39.065849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:39.065809    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:40.481367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:40.481320    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:40.605657 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:40.605618    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:40.759962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:40.759883    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:40.908962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:40.908916    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:41.298343 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:41.298307    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:41.513832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:41.513800    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:41.732321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:41.732287    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:42.515189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:42.515155    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:42.891413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:42.891377    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:44.339961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.339920    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9" podUID="4debb049-ebb2-4e5b-932a-04bfc8dd91c8" containerName="main" containerID="cri-o://24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e" gracePeriod=45
Nov 02 13:26:44.626618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.626589    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:26:44.644837 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.644807    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k" podUID="c5749f06-f474-4e93-b640-8eaeadc2ec35" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:26:44.725685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.725654    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:44.734493 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.734464    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"]
Nov 02 13:26:44.745536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.745465    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-kube-api-access-nk5rg\" (UniqueName: \"kubernetes.io/projected/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-kube-api-access-nk5rg\") pod \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") "
Nov 02 13:26:44.746192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.746169    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-home\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-home\") pod \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") "
Nov 02 13:26:44.746678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.746566    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-home" pod "4debb049-ebb2-4e5b-932a-04bfc8dd91c8" (UID: "4debb049-ebb2-4e5b-932a-04bfc8dd91c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:44.747148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.747131    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-dshm\") pod \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") "
Nov 02 13:26:44.747373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.747358    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tls-certs\") pod \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") "
Nov 02 13:26:44.747853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.747836    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tmp-dir\") pod \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") "
Nov 02 13:26:44.749108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.748151    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tmp-dir" pod "4debb049-ebb2-4e5b-932a-04bfc8dd91c8" (UID: "4debb049-ebb2-4e5b-932a-04bfc8dd91c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:44.749108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.748228    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-model-cache\") pod \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\" (UID: \"4debb049-ebb2-4e5b-932a-04bfc8dd91c8\") "
Nov 02 13:26:44.749108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.748488    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-model-cache" pod "4debb049-ebb2-4e5b-932a-04bfc8dd91c8" (UID: "4debb049-ebb2-4e5b-932a-04bfc8dd91c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:44.749108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.749050    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:44.749108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.749072    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:44.749108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.749087    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:44.752692 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.752655    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tls-certs" pod "4debb049-ebb2-4e5b-932a-04bfc8dd91c8" (UID: "4debb049-ebb2-4e5b-932a-04bfc8dd91c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:26:44.755225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.755201    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-dshm" pod "4debb049-ebb2-4e5b-932a-04bfc8dd91c8" (UID: "4debb049-ebb2-4e5b-932a-04bfc8dd91c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:26:44.760665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.760635    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-kube-api-access-nk5rg" pod "4debb049-ebb2-4e5b-932a-04bfc8dd91c8" (UID: "4debb049-ebb2-4e5b-932a-04bfc8dd91c8"). InnerVolumeSpecName "kube-api-access-nk5rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:44.857687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.856330    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk5rg\" (UniqueName: \"kubernetes.io/projected/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-kube-api-access-nk5rg\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:44.857687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.856364    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:44.857687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.856378    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4debb049-ebb2-4e5b-932a-04bfc8dd91c8-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:44.868017 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:26:44.867504    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:26:44.868017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:44.867940    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:45.127895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:45.127859    5879 generic.go:356] "Generic (PLEG): container finished" podID="4debb049-ebb2-4e5b-932a-04bfc8dd91c8" containerID="24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e" exitCode=0
Nov 02 13:26:45.128147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:45.127914    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"
Nov 02 13:26:45.128147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:45.127936    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9" event={"ID":"4debb049-ebb2-4e5b-932a-04bfc8dd91c8","Type":"ContainerDied","Data":"24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e"}
Nov 02 13:26:45.128147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:45.127975    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9" event={"ID":"4debb049-ebb2-4e5b-932a-04bfc8dd91c8","Type":"ContainerDied","Data":"7b43b2bc35979b601b8ab8c69b5c8e0b4a70223bfcfdb273dfe33f7c117948e2"}
Nov 02 13:26:45.128147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:45.128020    5879 scope.go:122] "RemoveContainer" containerID="24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e"
Nov 02 13:26:45.140294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:45.140264    5879 scope.go:122] "RemoveContainer" containerID="24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e"
Nov 02 13:26:45.140641 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:26:45.140620    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e\": container with ID starting with 24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e not found: ID does not exist" containerID="24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e"
Nov 02 13:26:45.140754 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:45.140647    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e"} err="failed to get container status \"24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e\": rpc error: code = NotFound desc = could not find container \"24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e\": container with ID starting with 24fcbb47d0ddcdffdbcd525d402b14903a31695e3ada50e7b31f03b47880da7e not found: ID does not exist"
Nov 02 13:26:45.199164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:45.199136    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"]
Nov 02 13:26:45.217720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:45.217690    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-23f720/test-model-23f720-kserve-7f4bbdcf75-5jpl9"]
Nov 02 13:26:45.275321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:45.275282    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:45.708807 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:45.708768    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:46.019270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:46.019189    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4debb049-ebb2-4e5b-932a-04bfc8dd91c8" path="/var/lib/kubelet/pods/4debb049-ebb2-4e5b-932a-04bfc8dd91c8/volumes"
Nov 02 13:26:46.031637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:46.031607    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"]
Nov 02 13:26:46.045575 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:46.045546    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-957c3d45-mdbg6"]
Nov 02 13:26:46.613257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:46.613228    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:48.018940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:48.018905    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52871ebe-9271-4d59-a2ac-e6c372c88eae" path="/var/lib/kubelet/pods/52871ebe-9271-4d59-a2ac-e6c372c88eae/volumes"
Nov 02 13:26:48.561824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:48.561788    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:48.707125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:48.707092    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"]
Nov 02 13:26:48.744874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:48.744846    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"]
Nov 02 13:26:48.818973 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:48.818900    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:48.945536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:48.945468    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:49.344389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:49.344313    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"
Nov 02 13:26:49.424601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:49.421856    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5749f06-f474-4e93-b640-8eaeadc2ec35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5749f06-f474-4e93-b640-8eaeadc2ec35-plugins-config-volume\") pod \"c5749f06-f474-4e93-b640-8eaeadc2ec35\" (UID: \"c5749f06-f474-4e93-b640-8eaeadc2ec35\") "
Nov 02 13:26:49.424601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:49.422022    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5749f06-f474-4e93-b640-8eaeadc2ec35-kube-api-access-tx442\" (UniqueName: \"kubernetes.io/projected/c5749f06-f474-4e93-b640-8eaeadc2ec35-kube-api-access-tx442\") pod \"c5749f06-f474-4e93-b640-8eaeadc2ec35\" (UID: \"c5749f06-f474-4e93-b640-8eaeadc2ec35\") "
Nov 02 13:26:49.426186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:49.425315    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5749f06-f474-4e93-b640-8eaeadc2ec35-plugins-config-volume" pod "c5749f06-f474-4e93-b640-8eaeadc2ec35" (UID: "c5749f06-f474-4e93-b640-8eaeadc2ec35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:26:49.460502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:49.460423    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:49.465901 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:49.465865    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5749f06-f474-4e93-b640-8eaeadc2ec35-kube-api-access-tx442" pod "c5749f06-f474-4e93-b640-8eaeadc2ec35" (UID: "c5749f06-f474-4e93-b640-8eaeadc2ec35"). InnerVolumeSpecName "kube-api-access-tx442". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:49.523814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:49.523786    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx442\" (UniqueName: \"kubernetes.io/projected/c5749f06-f474-4e93-b640-8eaeadc2ec35-kube-api-access-tx442\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:49.523814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:49.523814    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5749f06-f474-4e93-b640-8eaeadc2ec35-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:50.168934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:50.168901    5879 generic.go:356] "Generic (PLEG): container finished" podID="c5749f06-f474-4e93-b640-8eaeadc2ec35" containerID="3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d" exitCode=0
Nov 02 13:26:50.169209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:50.168954    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"
Nov 02 13:26:50.169209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:50.169005    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k" event={"ID":"c5749f06-f474-4e93-b640-8eaeadc2ec35","Type":"ContainerDied","Data":"3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d"}
Nov 02 13:26:50.169209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:50.169066    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k" event={"ID":"c5749f06-f474-4e93-b640-8eaeadc2ec35","Type":"ContainerDied","Data":"60697fe279a3d529cc8d928114fe678be1ddcc923242703c2f3d0e1387bd92a0"}
Nov 02 13:26:50.169209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:50.169096    5879 scope.go:122] "RemoveContainer" containerID="3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d"
Nov 02 13:26:50.181833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:50.181621    5879 scope.go:122] "RemoveContainer" containerID="3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d"
Nov 02 13:26:50.181934 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:26:50.181913    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d\": container with ID starting with 3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d not found: ID does not exist" containerID="3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d"
Nov 02 13:26:50.181979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:50.181945    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d"} err="failed to get container status \"3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d\": rpc error: code = NotFound desc = could not find container \"3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d\": container with ID starting with 3834faa3990a4b7c027d5cd7872c9ea47ae1dfbf74eec5b7dcaeb6fc1237379d not found: ID does not exist"
Nov 02 13:26:50.228758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:50.228732    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"]
Nov 02 13:26:50.242235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:50.242205    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e8a402-574cd44f6c-8dk2k"]
Nov 02 13:26:50.347790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:50.347702    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:52.021254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:52.021223    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5749f06-f474-4e93-b640-8eaeadc2ec35" path="/var/lib/kubelet/pods/c5749f06-f474-4e93-b640-8eaeadc2ec35/volumes"
Nov 02 13:26:52.110299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:52.110259    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"]
Nov 02 13:26:52.147563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:52.147529    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"]
Nov 02 13:26:52.261189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:52.261159    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:52.475018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:52.474970    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:53.019136 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:53.019104    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:53.708036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:53.707970    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk" podUID="cf747cbc-bea2-4edd-97d2-562ad019abb0" containerName="payload-processing" containerID="cri-o://d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075" gracePeriod=25
Nov 02 13:26:53.745632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:53.745598    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8" podUID="1eb2c475-7933-4442-8df2-91761ce122aa" containerName="payload-pre-processing" containerID="cri-o://a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395" gracePeriod=25
Nov 02 13:26:53.995822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:53.995794    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"
Nov 02 13:26:54.088883 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.088847    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf747cbc-bea2-4edd-97d2-562ad019abb0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf747cbc-bea2-4edd-97d2-562ad019abb0-plugins-config-volume\") pod \"cf747cbc-bea2-4edd-97d2-562ad019abb0\" (UID: \"cf747cbc-bea2-4edd-97d2-562ad019abb0\") "
Nov 02 13:26:54.089097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.089072    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf747cbc-bea2-4edd-97d2-562ad019abb0-kube-api-access-rmrhr\" (UniqueName: \"kubernetes.io/projected/cf747cbc-bea2-4edd-97d2-562ad019abb0-kube-api-access-rmrhr\") pod \"cf747cbc-bea2-4edd-97d2-562ad019abb0\" (UID: \"cf747cbc-bea2-4edd-97d2-562ad019abb0\") "
Nov 02 13:26:54.089268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.089233    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf747cbc-bea2-4edd-97d2-562ad019abb0-plugins-config-volume" pod "cf747cbc-bea2-4edd-97d2-562ad019abb0" (UID: "cf747cbc-bea2-4edd-97d2-562ad019abb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:26:54.089793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.089774    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf747cbc-bea2-4edd-97d2-562ad019abb0-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:54.091389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.091367    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf747cbc-bea2-4edd-97d2-562ad019abb0-kube-api-access-rmrhr" pod "cf747cbc-bea2-4edd-97d2-562ad019abb0" (UID: "cf747cbc-bea2-4edd-97d2-562ad019abb0"). InnerVolumeSpecName "kube-api-access-rmrhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:26:54.154909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.154880    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:54.191641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.191610    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmrhr\" (UniqueName: \"kubernetes.io/projected/cf747cbc-bea2-4edd-97d2-562ad019abb0-kube-api-access-rmrhr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:26:54.203864 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.203828    5879 generic.go:356] "Generic (PLEG): container finished" podID="cf747cbc-bea2-4edd-97d2-562ad019abb0" containerID="d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075" exitCode=0
Nov 02 13:26:54.204080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.203892    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"
Nov 02 13:26:54.204080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.203883    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk" event={"ID":"cf747cbc-bea2-4edd-97d2-562ad019abb0","Type":"ContainerDied","Data":"d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075"}
Nov 02 13:26:54.204080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.204063    5879 scope.go:122] "RemoveContainer" containerID="d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075"
Nov 02 13:26:54.207340 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.207310    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk" event={"ID":"cf747cbc-bea2-4edd-97d2-562ad019abb0","Type":"ContainerDied","Data":"5e859d763eea5ce57c3031c580536c738f8db86203f5596487f7752592d246af"}
Nov 02 13:26:54.218169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.218147    5879 scope.go:122] "RemoveContainer" containerID="d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075"
Nov 02 13:26:54.218533 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:26:54.218485    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075\": container with ID starting with d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075 not found: ID does not exist" containerID="d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075"
Nov 02 13:26:54.218533 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.218522    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075"} err="failed to get container status \"d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075\": rpc error: code = NotFound desc = could not find container \"d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075\": container with ID starting with d13111c665bbba344d5685f91eb388a38361138c768f8eeb38192a66be684075 not found: ID does not exist"
Nov 02 13:26:54.265162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.265123    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"]
Nov 02 13:26:54.268603 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:54.268579    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-357b30-6fbd545b95-vjnqk"]
Nov 02 13:26:55.899832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:55.899799    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:56.019866 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:56.019833    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf747cbc-bea2-4edd-97d2-562ad019abb0" path="/var/lib/kubelet/pods/cf747cbc-bea2-4edd-97d2-562ad019abb0/volumes"
Nov 02 13:26:56.267342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:56.267262    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:57.111438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:57.111398    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4" podUID="fdee021c-70d3-47c0-be6b-589d619ce6bc" containerName="payload-processing" containerID="cri-o://34f8909bbee0bd13282f52151111daebc2f49cc5e01c50b37aef31c83fea00af" gracePeriod=25
Nov 02 13:26:57.148882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:57.148843    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g" podUID="a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f" containerName="payload-pre-processing" containerID="cri-o://d6a5cb8977190f57036901d80f5e397c91bccbf2df77ed89a3aaf3d65eb9388b" gracePeriod=25
Nov 02 13:26:57.217885 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:57.217848    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8" podUID="1eb2c475-7933-4442-8df2-91761ce122aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:26:57.689314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:57.689256    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:59.482911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:59.482880    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:59.740844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:59.740750    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:26:59.763190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:26:59.763159    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw" podUID="c9327c2b-5304-4513-b21c-0f9174e60ff0" containerName="main" containerID="cri-o://850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c" gracePeriod=45
Nov 02 13:27:00.061826 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.061798    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:27:00.183254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.183218    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-model-cache\") pod \"c9327c2b-5304-4513-b21c-0f9174e60ff0\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") "
Nov 02 13:27:00.183447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.183298    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-tmp-dir\") pod \"c9327c2b-5304-4513-b21c-0f9174e60ff0\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") "
Nov 02 13:27:00.183447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.183350    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-dshm\") pod \"c9327c2b-5304-4513-b21c-0f9174e60ff0\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") "
Nov 02 13:27:00.183447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.183368    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-home\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-home\") pod \"c9327c2b-5304-4513-b21c-0f9174e60ff0\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") "
Nov 02 13:27:00.183447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.183416    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9327c2b-5304-4513-b21c-0f9174e60ff0-kube-api-access-xjfkd\" (UniqueName: \"kubernetes.io/projected/c9327c2b-5304-4513-b21c-0f9174e60ff0-kube-api-access-xjfkd\") pod \"c9327c2b-5304-4513-b21c-0f9174e60ff0\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") "
Nov 02 13:27:00.183711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.183497    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9327c2b-5304-4513-b21c-0f9174e60ff0-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9327c2b-5304-4513-b21c-0f9174e60ff0-tls-certs\") pod \"c9327c2b-5304-4513-b21c-0f9174e60ff0\" (UID: \"c9327c2b-5304-4513-b21c-0f9174e60ff0\") "
Nov 02 13:27:00.183711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.183549    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-model-cache" pod "c9327c2b-5304-4513-b21c-0f9174e60ff0" (UID: "c9327c2b-5304-4513-b21c-0f9174e60ff0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:00.183711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.183636    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-tmp-dir" pod "c9327c2b-5304-4513-b21c-0f9174e60ff0" (UID: "c9327c2b-5304-4513-b21c-0f9174e60ff0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:00.183711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.183678    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-home" pod "c9327c2b-5304-4513-b21c-0f9174e60ff0" (UID: "c9327c2b-5304-4513-b21c-0f9174e60ff0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:00.184133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.184113    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:00.184133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.184134    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:00.184304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.184146    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:00.185698 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.185679    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-dshm" pod "c9327c2b-5304-4513-b21c-0f9174e60ff0" (UID: "c9327c2b-5304-4513-b21c-0f9174e60ff0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:27:00.186286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.186267    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9327c2b-5304-4513-b21c-0f9174e60ff0-tls-certs" pod "c9327c2b-5304-4513-b21c-0f9174e60ff0" (UID: "c9327c2b-5304-4513-b21c-0f9174e60ff0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:27:00.186366 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.186332    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9327c2b-5304-4513-b21c-0f9174e60ff0-kube-api-access-xjfkd" pod "c9327c2b-5304-4513-b21c-0f9174e60ff0" (UID: "c9327c2b-5304-4513-b21c-0f9174e60ff0"). InnerVolumeSpecName "kube-api-access-xjfkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:00.259128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.259093    5879 generic.go:356] "Generic (PLEG): container finished" podID="c9327c2b-5304-4513-b21c-0f9174e60ff0" containerID="850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c" exitCode=0
Nov 02 13:27:00.259371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.259147    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"
Nov 02 13:27:00.259371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.259188    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw" event={"ID":"c9327c2b-5304-4513-b21c-0f9174e60ff0","Type":"ContainerDied","Data":"850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c"}
Nov 02 13:27:00.259371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.259244    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw" event={"ID":"c9327c2b-5304-4513-b21c-0f9174e60ff0","Type":"ContainerDied","Data":"5f8be04e742a3303b0e611df9abb24acfa4330c1dbd572b127abc83f3fc73e37"}
Nov 02 13:27:00.259371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.259268    5879 scope.go:122] "RemoveContainer" containerID="850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c"
Nov 02 13:27:00.270873 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.270856    5879 scope.go:122] "RemoveContainer" containerID="850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c"
Nov 02 13:27:00.271163 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:27:00.271142    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c\": container with ID starting with 850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c not found: ID does not exist" containerID="850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c"
Nov 02 13:27:00.271245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.271170    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c"} err="failed to get container status \"850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c\": rpc error: code = NotFound desc = could not find container \"850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c\": container with ID starting with 850a8cd3485bb42e1a1e8870a6ca96704fe9fc855e4bf0bdcb43ad4ac6ac7b5c not found: ID does not exist"
Nov 02 13:27:00.285937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.285867    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9327c2b-5304-4513-b21c-0f9174e60ff0-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:00.285937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.285890    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9327c2b-5304-4513-b21c-0f9174e60ff0-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:00.285937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.285900    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjfkd\" (UniqueName: \"kubernetes.io/projected/c9327c2b-5304-4513-b21c-0f9174e60ff0-kube-api-access-xjfkd\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:00.319035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.319012    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"]
Nov 02 13:27:00.331488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:00.331463    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8962e9/test-model-8962e9-kserve-64674c5bc7-5mpbw"]
Nov 02 13:27:01.170962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:01.170922    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:02.017945 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:02.017913    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9327c2b-5304-4513-b21c-0f9174e60ff0" path="/var/lib/kubelet/pods/c9327c2b-5304-4513-b21c-0f9174e60ff0/volumes"
Nov 02 13:27:02.892778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:02.892747    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:03.722019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:03.721580    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:04.118112 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:04.118074    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g" podUID="a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:27:04.123015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:04.121181    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4" podUID="fdee021c-70d3-47c0-be6b-589d619ce6bc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:27:05.556787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:05.556724    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:06.007473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:06.007420    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:06.169570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:06.169536    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:06.748192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:06.748158    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:07.137629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:07.137576    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:07.236670 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:07.236628    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8" podUID="1eb2c475-7933-4442-8df2-91761ce122aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:27:07.587954 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:07.587859    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:07.934316 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:07.934273    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:09.388637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:09.388610    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:11.120829 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:11.120794    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:12.325703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:12.325664    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:12.441803 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:12.441772    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:12.933602 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:12.933562    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:14.024190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.024127    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"
Nov 02 13:27:14.101143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.101097    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4" podUID="fdee021c-70d3-47c0-be6b-589d619ce6bc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:27:14.101354 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.101113    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g" podUID="a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:27:14.114335 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.114305    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb2c475-7933-4442-8df2-91761ce122aa-kube-api-access-8djht\" (UniqueName: \"kubernetes.io/projected/1eb2c475-7933-4442-8df2-91761ce122aa-kube-api-access-8djht\") pod \"1eb2c475-7933-4442-8df2-91761ce122aa\" (UID: \"1eb2c475-7933-4442-8df2-91761ce122aa\") "
Nov 02 13:27:14.114519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.114505    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb2c475-7933-4442-8df2-91761ce122aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb2c475-7933-4442-8df2-91761ce122aa-plugins-config-volume\") pod \"1eb2c475-7933-4442-8df2-91761ce122aa\" (UID: \"1eb2c475-7933-4442-8df2-91761ce122aa\") "
Nov 02 13:27:14.115008 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.114948    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb2c475-7933-4442-8df2-91761ce122aa-plugins-config-volume" pod "1eb2c475-7933-4442-8df2-91761ce122aa" (UID: "1eb2c475-7933-4442-8df2-91761ce122aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:27:14.116721 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.116698    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb2c475-7933-4442-8df2-91761ce122aa-kube-api-access-8djht" pod "1eb2c475-7933-4442-8df2-91761ce122aa" (UID: "1eb2c475-7933-4442-8df2-91761ce122aa"). InnerVolumeSpecName "kube-api-access-8djht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:14.216405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.216373    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8djht\" (UniqueName: \"kubernetes.io/projected/1eb2c475-7933-4442-8df2-91761ce122aa-kube-api-access-8djht\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:14.216405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.216401    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb2c475-7933-4442-8df2-91761ce122aa-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:14.405338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.405293    5879 generic.go:356] "Generic (PLEG): container finished" podID="1eb2c475-7933-4442-8df2-91761ce122aa" containerID="a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395" exitCode=0
Nov 02 13:27:14.405535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.405345    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"
Nov 02 13:27:14.405535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.405393    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8" event={"ID":"1eb2c475-7933-4442-8df2-91761ce122aa","Type":"ContainerDied","Data":"a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395"}
Nov 02 13:27:14.405535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.405446    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8" event={"ID":"1eb2c475-7933-4442-8df2-91761ce122aa","Type":"ContainerDied","Data":"192c9509b2721e1129714b7293af5703304c5c1771f457cbc81c43cd81a6f09c"}
Nov 02 13:27:14.405535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.405472    5879 scope.go:122] "RemoveContainer" containerID="a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395"
Nov 02 13:27:14.416839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.416823    5879 scope.go:122] "RemoveContainer" containerID="a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395"
Nov 02 13:27:14.417153 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:27:14.417130    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395\": container with ID starting with a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395 not found: ID does not exist" containerID="a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395"
Nov 02 13:27:14.417264 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.417159    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395"} err="failed to get container status \"a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395\": rpc error: code = NotFound desc = could not find container \"a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395\": container with ID starting with a9e85a2e0c153149adc45ebdbe992d7234271221eb41930bcbf39c78acc5e395 not found: ID does not exist"
Nov 02 13:27:14.464278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.464246    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"]
Nov 02 13:27:14.467555 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.467531    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-357b30-b4f4f9d7-dl8q8"]
Nov 02 13:27:14.661788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:14.661667    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:14.822641 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:27:14.822605    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:27:15.057218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:15.057137    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"]
Nov 02 13:27:15.091747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:15.091711    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"]
Nov 02 13:27:15.229541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:15.229483    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:15.368394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:15.368365    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:15.493439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:15.493399    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:16.018717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:16.018678    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb2c475-7933-4442-8df2-91761ce122aa" path="/var/lib/kubelet/pods/1eb2c475-7933-4442-8df2-91761ce122aa/volumes"
Nov 02 13:27:16.459979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:16.459947    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:17.432069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.432040    5879 generic.go:356] "Generic (PLEG): container finished" podID="fdee021c-70d3-47c0-be6b-589d619ce6bc" containerID="34f8909bbee0bd13282f52151111daebc2f49cc5e01c50b37aef31c83fea00af" exitCode=0
Nov 02 13:27:17.432228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.432117    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4" event={"ID":"fdee021c-70d3-47c0-be6b-589d619ce6bc","Type":"ContainerDied","Data":"34f8909bbee0bd13282f52151111daebc2f49cc5e01c50b37aef31c83fea00af"}
Nov 02 13:27:17.432228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.432177    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4" event={"ID":"fdee021c-70d3-47c0-be6b-589d619ce6bc","Type":"ContainerDied","Data":"afbe09d8b17be228d8d1fc313768975a26ce15d5272c22f33ce730f5f82525aa"}
Nov 02 13:27:17.432228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.432201    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afbe09d8b17be228d8d1fc313768975a26ce15d5272c22f33ce730f5f82525aa"
Nov 02 13:27:17.433847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.433810    5879 generic.go:356] "Generic (PLEG): container finished" podID="a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f" containerID="d6a5cb8977190f57036901d80f5e397c91bccbf2df77ed89a3aaf3d65eb9388b" exitCode=0
Nov 02 13:27:17.433962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.433879    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g" event={"ID":"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f","Type":"ContainerDied","Data":"d6a5cb8977190f57036901d80f5e397c91bccbf2df77ed89a3aaf3d65eb9388b"}
Nov 02 13:27:17.435808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.435793    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"
Nov 02 13:27:17.477864 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.477837    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"
Nov 02 13:27:17.562299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.562266    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-kube-api-access-gvhxf\" (UniqueName: \"kubernetes.io/projected/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-kube-api-access-gvhxf\") pod \"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f\" (UID: \"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f\") "
Nov 02 13:27:17.562474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.562371    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-plugins-config-volume\") pod \"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f\" (UID: \"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f\") "
Nov 02 13:27:17.562474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.562438    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdee021c-70d3-47c0-be6b-589d619ce6bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdee021c-70d3-47c0-be6b-589d619ce6bc-plugins-config-volume\") pod \"fdee021c-70d3-47c0-be6b-589d619ce6bc\" (UID: \"fdee021c-70d3-47c0-be6b-589d619ce6bc\") "
Nov 02 13:27:17.562615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.562532    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdee021c-70d3-47c0-be6b-589d619ce6bc-kube-api-access-mbfx9\" (UniqueName: \"kubernetes.io/projected/fdee021c-70d3-47c0-be6b-589d619ce6bc-kube-api-access-mbfx9\") pod \"fdee021c-70d3-47c0-be6b-589d619ce6bc\" (UID: \"fdee021c-70d3-47c0-be6b-589d619ce6bc\") "
Nov 02 13:27:17.562811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.562784    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-plugins-config-volume" pod "a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f" (UID: "a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:27:17.562899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.562835    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdee021c-70d3-47c0-be6b-589d619ce6bc-plugins-config-volume" pod "fdee021c-70d3-47c0-be6b-589d619ce6bc" (UID: "fdee021c-70d3-47c0-be6b-589d619ce6bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:27:17.563366 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.563339    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:17.563478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.563372    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdee021c-70d3-47c0-be6b-589d619ce6bc-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:17.564912 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.564886    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdee021c-70d3-47c0-be6b-589d619ce6bc-kube-api-access-mbfx9" pod "fdee021c-70d3-47c0-be6b-589d619ce6bc" (UID: "fdee021c-70d3-47c0-be6b-589d619ce6bc"). InnerVolumeSpecName "kube-api-access-mbfx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:17.565038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.564926    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-kube-api-access-gvhxf" pod "a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f" (UID: "a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f"). InnerVolumeSpecName "kube-api-access-gvhxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:17.664810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.664705    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhxf\" (UniqueName: \"kubernetes.io/projected/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f-kube-api-access-gvhxf\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:17.664810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:17.664746    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbfx9\" (UniqueName: \"kubernetes.io/projected/fdee021c-70d3-47c0-be6b-589d619ce6bc-kube-api-access-mbfx9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:18.393166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:18.393133    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:18.446574 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:18.446537    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"
Nov 02 13:27:18.446574 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:18.446553    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g" event={"ID":"a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f","Type":"ContainerDied","Data":"4a380114ab41b7e7eb58993dfeb849636a3448d6f2645cf8c676bd455f8cf987"}
Nov 02 13:27:18.446845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:18.446616    5879 scope.go:122] "RemoveContainer" containerID="d6a5cb8977190f57036901d80f5e397c91bccbf2df77ed89a3aaf3d65eb9388b"
Nov 02 13:27:18.446845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:18.446556    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"
Nov 02 13:27:18.505606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:18.505575    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"]
Nov 02 13:27:18.519387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:18.519357    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-23f720-5684f8ccbc-gmk4g"]
Nov 02 13:27:18.557367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:18.557339    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"]
Nov 02 13:27:18.560550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:18.560522    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-23f720-7cd49f58f-pmdm4"]
Nov 02 13:27:20.020921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.020877    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f" path="/var/lib/kubelet/pods/a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f/volumes"
Nov 02 13:27:20.021768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.021743    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdee021c-70d3-47c0-be6b-589d619ce6bc" path="/var/lib/kubelet/pods/fdee021c-70d3-47c0-be6b-589d619ce6bc/volumes"
Nov 02 13:27:20.057501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.057431    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z" podUID="9a8dd290-b517-4712-a174-8e1959783e80" containerName="payload-processing" containerID="cri-o://a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5" gracePeriod=25
Nov 02 13:27:20.058447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.058400    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:20.093059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.092974    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc" podUID="1cb1b247-542c-4faa-a654-d6de0ab9b19c" containerName="payload-pre-processing" containerID="cri-o://6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad" gracePeriod=25
Nov 02 13:27:20.329379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.329354    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"
Nov 02 13:27:20.463677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.463639    5879 generic.go:356] "Generic (PLEG): container finished" podID="9a8dd290-b517-4712-a174-8e1959783e80" containerID="a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5" exitCode=0
Nov 02 13:27:20.463861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.463689    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"
Nov 02 13:27:20.463861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.463728    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z" event={"ID":"9a8dd290-b517-4712-a174-8e1959783e80","Type":"ContainerDied","Data":"a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5"}
Nov 02 13:27:20.463861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.463781    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z" event={"ID":"9a8dd290-b517-4712-a174-8e1959783e80","Type":"ContainerDied","Data":"e4f22368b673b1fb7d15c55977a6f21e6e3cf89fde292738a316748b9319fa21"}
Nov 02 13:27:20.463861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.463803    5879 scope.go:122] "RemoveContainer" containerID="a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5"
Nov 02 13:27:20.474558 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.474538    5879 scope.go:122] "RemoveContainer" containerID="a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5"
Nov 02 13:27:20.474841 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:27:20.474821    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5\": container with ID starting with a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5 not found: ID does not exist" containerID="a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5"
Nov 02 13:27:20.474901 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.474851    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5"} err="failed to get container status \"a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5\": rpc error: code = NotFound desc = could not find container \"a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5\": container with ID starting with a85608f60cfde7093239a1641ebe768463c9b3d89006ae60df7ac58885ce94c5 not found: ID does not exist"
Nov 02 13:27:20.507417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.507389    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8dd290-b517-4712-a174-8e1959783e80-kube-api-access-bqhnn\" (UniqueName: \"kubernetes.io/projected/9a8dd290-b517-4712-a174-8e1959783e80-kube-api-access-bqhnn\") pod \"9a8dd290-b517-4712-a174-8e1959783e80\" (UID: \"9a8dd290-b517-4712-a174-8e1959783e80\") "
Nov 02 13:27:20.507563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.507506    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8dd290-b517-4712-a174-8e1959783e80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8dd290-b517-4712-a174-8e1959783e80-plugins-config-volume\") pod \"9a8dd290-b517-4712-a174-8e1959783e80\" (UID: \"9a8dd290-b517-4712-a174-8e1959783e80\") "
Nov 02 13:27:20.507797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.507768    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8dd290-b517-4712-a174-8e1959783e80-plugins-config-volume" pod "9a8dd290-b517-4712-a174-8e1959783e80" (UID: "9a8dd290-b517-4712-a174-8e1959783e80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:27:20.508140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.508109    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a8dd290-b517-4712-a174-8e1959783e80-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:20.509694 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.509672    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8dd290-b517-4712-a174-8e1959783e80-kube-api-access-bqhnn" pod "9a8dd290-b517-4712-a174-8e1959783e80" (UID: "9a8dd290-b517-4712-a174-8e1959783e80"). InnerVolumeSpecName "kube-api-access-bqhnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:20.609531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.609450    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqhnn\" (UniqueName: \"kubernetes.io/projected/9a8dd290-b517-4712-a174-8e1959783e80-kube-api-access-bqhnn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:20.813942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.813908    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"]
Nov 02 13:27:20.817282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:20.817261    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8962e9-59cc45b88f-mgc8z"]
Nov 02 13:27:21.759244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:21.759209    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc" podUID="1cb1b247-542c-4faa-a654-d6de0ab9b19c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:27:21.803124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:21.803093    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:22.018358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:22.018279    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8dd290-b517-4712-a174-8e1959783e80" path="/var/lib/kubelet/pods/9a8dd290-b517-4712-a174-8e1959783e80/volumes"
Nov 02 13:27:22.270955 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:22.270851    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:23.672978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:23.672942    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:25.350784 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:25.350740    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:27.068409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:27.068375    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:28.780675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:28.780636    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:30.300964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:30.300922    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:30.970465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:30.970348    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:31.759474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:31.759425    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc" podUID="1cb1b247-542c-4faa-a654-d6de0ab9b19c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:27:32.785370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:32.785302    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:32.912282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:32.912239    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:34.412276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:34.412239    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:34.541180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:34.541132    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:35.244208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:35.244175    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:35.662704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:35.662674    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:36.616875 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:36.616842    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:37.820600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:37.820561    5879 scope.go:122] "RemoveContainer" containerID="28b2663860d3efb7c385c53fb1b0f4a09dcf3e05b6214aefda7d878a65a88ba0"
Nov 02 13:27:37.832470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:37.832443    5879 scope.go:122] "RemoveContainer" containerID="93977a7222d62e3200fd8aca70b74ea5991970a009f55b6b341fb3771074ee71"
Nov 02 13:27:37.844314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:37.844287    5879 scope.go:122] "RemoveContainer" containerID="f80bd6e5ff469447517e76f814de3bb949db5b95d99f30885a7742930053aa04"
Nov 02 13:27:37.854367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:37.854343    5879 scope.go:122] "RemoveContainer" containerID="efd24be06fb45985544760c8ec48a3123254794da7f2500d738314d6dc70203c"
Nov 02 13:27:37.865055 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:37.865034    5879 scope.go:122] "RemoveContainer" containerID="34f8909bbee0bd13282f52151111daebc2f49cc5e01c50b37aef31c83fea00af"
Nov 02 13:27:38.482906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.482854    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:38.826951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.826874    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"]
Nov 02 13:27:38.827687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827670    5879 cpu_manager.go:414] "containerMap: removing container" podUID="a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f" containerName="payload-pre-processing"
Nov 02 13:27:38.827752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827689    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f" containerName="payload-pre-processing"
Nov 02 13:27:38.827752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827704    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae" containerName="revoke-keys"
Nov 02 13:27:38.827752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827709    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae" containerName="revoke-keys"
Nov 02 13:27:38.827752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827717    5879 cpu_manager.go:414] "containerMap: removing container" podUID="cf747cbc-bea2-4edd-97d2-562ad019abb0" containerName="payload-processing"
Nov 02 13:27:38.827752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827722    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf747cbc-bea2-4edd-97d2-562ad019abb0" containerName="payload-processing"
Nov 02 13:27:38.827752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827743    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c5749f06-f474-4e93-b640-8eaeadc2ec35" containerName="payload-pre-processing"
Nov 02 13:27:38.827752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827748    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5749f06-f474-4e93-b640-8eaeadc2ec35" containerName="payload-pre-processing"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827756    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9a8dd290-b517-4712-a174-8e1959783e80" containerName="payload-processing"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827761    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8dd290-b517-4712-a174-8e1959783e80" containerName="payload-processing"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827773    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1eb2c475-7933-4442-8df2-91761ce122aa" containerName="payload-pre-processing"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827781    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb2c475-7933-4442-8df2-91761ce122aa" containerName="payload-pre-processing"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827802    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4debb049-ebb2-4e5b-932a-04bfc8dd91c8" containerName="main"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827810    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4debb049-ebb2-4e5b-932a-04bfc8dd91c8" containerName="main"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827822    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c9327c2b-5304-4513-b21c-0f9174e60ff0" containerName="main"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827827    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9327c2b-5304-4513-b21c-0f9174e60ff0" containerName="main"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827838    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4aa69de3-1d00-44f8-af1e-796146e70910" containerName="main"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827843    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa69de3-1d00-44f8-af1e-796146e70910" containerName="main"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827849    5879 cpu_manager.go:414] "containerMap: removing container" podUID="fdee021c-70d3-47c0-be6b-589d619ce6bc" containerName="payload-processing"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827853    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdee021c-70d3-47c0-be6b-589d619ce6bc" containerName="payload-processing"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827865    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5e2e31b0-7ac4-45a3-a927-0670eea8e6df" containerName="revoke-keys"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827870    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2e31b0-7ac4-45a3-a927-0670eea8e6df" containerName="revoke-keys"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.827986    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e2e31b0-7ac4-45a3-a927-0670eea8e6df" containerName="revoke-keys"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.828022    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22afe41-b0e8-4ef4-8ce2-50d5ba0d4c1f" containerName="payload-pre-processing"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.828031    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae" containerName="revoke-keys"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.828039    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf747cbc-bea2-4edd-97d2-562ad019abb0" containerName="payload-processing"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.828055    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5749f06-f474-4e93-b640-8eaeadc2ec35" containerName="payload-pre-processing"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.828062    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8dd290-b517-4712-a174-8e1959783e80" containerName="payload-processing"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.828072    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb2c475-7933-4442-8df2-91761ce122aa" containerName="payload-pre-processing"
Nov 02 13:27:38.828070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.828084    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4debb049-ebb2-4e5b-932a-04bfc8dd91c8" containerName="main"
Nov 02 13:27:38.828763 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.828095    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9327c2b-5304-4513-b21c-0f9174e60ff0" containerName="main"
Nov 02 13:27:38.828763 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.828106    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa69de3-1d00-44f8-af1e-796146e70910" containerName="main"
Nov 02 13:27:38.828763 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.828116    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdee021c-70d3-47c0-be6b-589d619ce6bc" containerName="payload-processing"
Nov 02 13:27:38.836618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.834300    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:38.854772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.854742    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"]
Nov 02 13:27:38.861571 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.861540    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8032f3\"/\"openshift-service-ca.crt\""
Nov 02 13:27:38.861767 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.861633    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-8032f3\"/\"kube-root-ca.crt\""
Nov 02 13:27:38.861767 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.861693    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8032f3\"/\"rate-test-model-8032f3-kserve-self-signed-certs\""
Nov 02 13:27:38.869940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.869905    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-8032f3\"/\"default-dockercfg-28fjx\""
Nov 02 13:27:38.904238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.904194    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-model-cache\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:38.904441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.904268    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tmp-dir\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:38.904441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.904335    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-dshm\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:38.904441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.904355    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tls-certs\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:38.904441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.904408    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-home\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:38.904695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.904483    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67hhd\" (UniqueName: \"kubernetes.io/projected/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-kube-api-access-67hhd\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:38.912765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:38.912724    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:39.005728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.005690    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-model-cache\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.005958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.005769    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tmp-dir\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.005958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.005812    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-dshm\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.005958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.005828    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tls-certs\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.005958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.005899    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-home\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.005958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.005936    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67hhd\" (UniqueName: \"kubernetes.io/projected/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-kube-api-access-67hhd\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.006321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.006172    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-model-cache\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.006321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.006232    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tmp-dir\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.006491 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.006470    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-home\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.008486 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.008451    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-dshm\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.008657 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.008637    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tls-certs\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.031914 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.031884    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67hhd\" (UniqueName: \"kubernetes.io/projected/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-kube-api-access-67hhd\") pod \"rate-test-model-8032f3-kserve-758c77fd5c-xrlr8\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") " pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.159325 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.159276    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:39.504856 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:27:39.504822    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe7c00e6_5bde_4c59_bda7_6ea5894cabfb.slice/crio-fa76baf2ab1ebdd556e0246a028deb57253eefa72295a1417b00a1043cd12fae WatchSource:0}: Error finding container fa76baf2ab1ebdd556e0246a028deb57253eefa72295a1417b00a1043cd12fae: Status 404 returned error can't find the container with id fa76baf2ab1ebdd556e0246a028deb57253eefa72295a1417b00a1043cd12fae
Nov 02 13:27:39.505513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.505486    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"]
Nov 02 13:27:39.622909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.622843    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8" event={"ID":"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb","Type":"ContainerStarted","Data":"3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d"}
Nov 02 13:27:39.622909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.622896    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8" event={"ID":"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb","Type":"ContainerStarted","Data":"fa76baf2ab1ebdd556e0246a028deb57253eefa72295a1417b00a1043cd12fae"}
Nov 02 13:27:39.702446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:39.702391    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8" podStartSLOduration=1.702376903 podStartE2EDuration="1.702376903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:27:39.69881152 +0000 UTC m=+3605.805514372" watchObservedRunningTime="2026-11-02 13:27:39.702376903 +0000 UTC m=+3605.809079783"
Nov 02 13:27:40.159467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.159427    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:40.379448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.379412    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"
Nov 02 13:27:40.428169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.428133    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb1b247-542c-4faa-a654-d6de0ab9b19c-kube-api-access-8gg5s\" (UniqueName: \"kubernetes.io/projected/1cb1b247-542c-4faa-a654-d6de0ab9b19c-kube-api-access-8gg5s\") pod \"1cb1b247-542c-4faa-a654-d6de0ab9b19c\" (UID: \"1cb1b247-542c-4faa-a654-d6de0ab9b19c\") "
Nov 02 13:27:40.428389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.428246    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb1b247-542c-4faa-a654-d6de0ab9b19c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb1b247-542c-4faa-a654-d6de0ab9b19c-plugins-config-volume\") pod \"1cb1b247-542c-4faa-a654-d6de0ab9b19c\" (UID: \"1cb1b247-542c-4faa-a654-d6de0ab9b19c\") "
Nov 02 13:27:40.428894 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.428861    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb1b247-542c-4faa-a654-d6de0ab9b19c-plugins-config-volume" pod "1cb1b247-542c-4faa-a654-d6de0ab9b19c" (UID: "1cb1b247-542c-4faa-a654-d6de0ab9b19c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:27:40.431102 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.431072    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb1b247-542c-4faa-a654-d6de0ab9b19c-kube-api-access-8gg5s" pod "1cb1b247-542c-4faa-a654-d6de0ab9b19c" (UID: "1cb1b247-542c-4faa-a654-d6de0ab9b19c"). InnerVolumeSpecName "kube-api-access-8gg5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:27:40.467370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.467326    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:40.530293 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.530256    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gg5s\" (UniqueName: \"kubernetes.io/projected/1cb1b247-542c-4faa-a654-d6de0ab9b19c-kube-api-access-8gg5s\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:40.530293 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.530285    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cb1b247-542c-4faa-a654-d6de0ab9b19c-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:27:40.646329 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.646290    5879 generic.go:356] "Generic (PLEG): container finished" podID="1cb1b247-542c-4faa-a654-d6de0ab9b19c" containerID="6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad" exitCode=0
Nov 02 13:27:40.646526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.646355    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"
Nov 02 13:27:40.646526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.646369    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc" event={"ID":"1cb1b247-542c-4faa-a654-d6de0ab9b19c","Type":"ContainerDied","Data":"6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad"}
Nov 02 13:27:40.646526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.646454    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc" event={"ID":"1cb1b247-542c-4faa-a654-d6de0ab9b19c","Type":"ContainerDied","Data":"6cb25c139c4786e9aa67867e68a5663aa8be4cccfd557fbb2f6ca58983ef1bb1"}
Nov 02 13:27:40.646526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.646478    5879 scope.go:122] "RemoveContainer" containerID="6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad"
Nov 02 13:27:40.665620 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.665601    5879 scope.go:122] "RemoveContainer" containerID="6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad"
Nov 02 13:27:40.665925 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:27:40.665902    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad\": container with ID starting with 6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad not found: ID does not exist" containerID="6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad"
Nov 02 13:27:40.666034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.665933    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad"} err="failed to get container status \"6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad\": rpc error: code = NotFound desc = could not find container \"6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad\": container with ID starting with 6bd5b0fc08b5824e9126b644ae46ba241a8bd448e6814ac1af970afb0e19d0ad not found: ID does not exist"
Nov 02 13:27:40.705047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.705013    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"]
Nov 02 13:27:40.718780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.718746    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8962e9-5bf5967579-6f6nc"]
Nov 02 13:27:40.896399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:40.896312    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:42.021632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:42.021595    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb1b247-542c-4faa-a654-d6de0ab9b19c" path="/var/lib/kubelet/pods/1cb1b247-542c-4faa-a654-d6de0ab9b19c/volumes"
Nov 02 13:27:42.318839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:42.318755    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:44.006431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:44.006351    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:44.823332 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:27:44.823297    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:27:45.780658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:45.780594    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:47.575729 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:47.575683    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:49.160004 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:49.159953    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:49.173882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:49.173848    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:49.549940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:49.549856    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:49.727577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:49.727546    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:27:51.551406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:51.551365    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:53.474465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:53.474427    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:55.377483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:55.377447    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:57.278887 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:57.278844    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:58.869432 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:58.869390    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:27:59.590157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:27:59.590113    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:00.492941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:00.492904    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:00.620897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:00.620865    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:01.363272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:01.363233    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:01.575309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:01.575273    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:01.713627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:01.713585    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"]
Nov 02 13:28:01.714056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:01.713966    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy" containerID="cri-o://d4ea8c0870cb2c9ee6524121127a3ba3eab870238c7f75bc51110f0abd1d4871" gracePeriod=30
Nov 02 13:28:01.742240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:01.742202    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:03.366789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.366750    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"]
Nov 02 13:28:03.367805 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.367780    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1cb1b247-542c-4faa-a654-d6de0ab9b19c" containerName="payload-pre-processing"
Nov 02 13:28:03.367805 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.367806    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb1b247-542c-4faa-a654-d6de0ab9b19c" containerName="payload-pre-processing"
Nov 02 13:28:03.368079 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.368064    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb1b247-542c-4faa-a654-d6de0ab9b19c" containerName="payload-pre-processing"
Nov 02 13:28:03.375029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.374115    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.396085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.391008    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"]
Nov 02 13:28:03.396085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.395238    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a6f2c0\"/\"kube-root-ca.crt\""
Nov 02 13:28:03.396085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.395342    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a6f2c0\"/\"rate-test-model-a6f2c0-kserve-self-signed-certs\""
Nov 02 13:28:03.411421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.405222    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a6f2c0\"/\"openshift-service-ca.crt\""
Nov 02 13:28:03.411421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.405350    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a6f2c0\"/\"default-dockercfg-vf8k9\""
Nov 02 13:28:03.427332 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.427288    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:03.488657 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.488610    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-model-cache\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.488657 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.488664    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-dshm\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.488942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.488764    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tls-certs\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.488942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.488874    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tmp-dir\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.488942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.488912    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-home\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.489152 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.489014    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fk9\" (UniqueName: \"kubernetes.io/projected/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-kube-api-access-22fk9\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.591971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.590766    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-home\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.591971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.590848    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fk9\" (UniqueName: \"kubernetes.io/projected/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-kube-api-access-22fk9\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.591971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.591187    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-model-cache\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.591971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.591231    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-dshm\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.591971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.591423    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tls-certs\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.591971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.591430    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-home\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.591971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.591626    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tmp-dir\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.592534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.592086    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tmp-dir\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.592534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.592340    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-model-cache\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.618057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.599603    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-dshm\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.618341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.618299    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tls-certs\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.621620 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.621588    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fk9\" (UniqueName: \"kubernetes.io/projected/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-kube-api-access-22fk9\") pod \"rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") " pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.704197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.704157    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:03.713276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:03.713240    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:04.037847 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:28:04.037815    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2c9b6a2_cc01_445d_9f9f_b9fbcd7776ed.slice/crio-cc47927f909c7d742b4318f3c7ce54e3273e979d7f9844724df08b385069ab34 WatchSource:0}: Error finding container cc47927f909c7d742b4318f3c7ce54e3273e979d7f9844724df08b385069ab34: Status 404 returned error can't find the container with id cc47927f909c7d742b4318f3c7ce54e3273e979d7f9844724df08b385069ab34
Nov 02 13:28:04.039141 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:04.039116    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"]
Nov 02 13:28:04.876106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:04.876067    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2" event={"ID":"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed","Type":"ContainerStarted","Data":"7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61"}
Nov 02 13:28:04.876106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:04.876115    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2" event={"ID":"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed","Type":"ContainerStarted","Data":"cc47927f909c7d742b4318f3c7ce54e3273e979d7f9844724df08b385069ab34"}
Nov 02 13:28:04.923418 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:04.923354    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2" podStartSLOduration=1.9233373679999999 podStartE2EDuration="1.923337368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:28:04.9204739 +0000 UTC m=+3631.027176751" watchObservedRunningTime="2026-11-02 13:28:04.923337368 +0000 UTC m=+3631.030040219"
Nov 02 13:28:05.675891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:05.675860    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:05.704910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:05.704873    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:06.895398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:06.895359    5879 generic.go:356] "Generic (PLEG): container finished" podID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerID="d4ea8c0870cb2c9ee6524121127a3ba3eab870238c7f75bc51110f0abd1d4871" exitCode=0
Nov 02 13:28:06.896391 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:06.896364    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" event={"ID":"f942a798-a6e7-427b-a16f-a6ed500bfa61","Type":"ContainerDied","Data":"d4ea8c0870cb2c9ee6524121127a3ba3eab870238c7f75bc51110f0abd1d4871"}
Nov 02 13:28:06.896522 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:06.896471    5879 scope.go:122] "RemoveContainer" containerID="67f967640efc2e64354aa65138ec13065d587fcec2c13d84b3a0b781540f32f0"
Nov 02 13:28:07.008552 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.008480    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:28:07.163644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.163605    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-certs\") pod \"f942a798-a6e7-427b-a16f-a6ed500bfa61\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") "
Nov 02 13:28:07.163852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.163664    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-data\") pod \"f942a798-a6e7-427b-a16f-a6ed500bfa61\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") "
Nov 02 13:28:07.163852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.163818    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-socket\") pod \"f942a798-a6e7-427b-a16f-a6ed500bfa61\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") "
Nov 02 13:28:07.163986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.163874    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-credential-socket\") pod \"f942a798-a6e7-427b-a16f-a6ed500bfa61\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") "
Nov 02 13:28:07.163986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.163957    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-certs" pod "f942a798-a6e7-427b-a16f-a6ed500bfa61" (UID: "f942a798-a6e7-427b-a16f-a6ed500bfa61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:07.164128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164020    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-envoy\") pod \"f942a798-a6e7-427b-a16f-a6ed500bfa61\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") "
Nov 02 13:28:07.164128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164048    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f942a798-a6e7-427b-a16f-a6ed500bfa61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942a798-a6e7-427b-a16f-a6ed500bfa61-istiod-ca-cert\") pod \"f942a798-a6e7-427b-a16f-a6ed500bfa61\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") "
Nov 02 13:28:07.164128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164051    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-data" pod "f942a798-a6e7-427b-a16f-a6ed500bfa61" (UID: "f942a798-a6e7-427b-a16f-a6ed500bfa61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:07.164128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164112    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-podinfo\") pod \"f942a798-a6e7-427b-a16f-a6ed500bfa61\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") "
Nov 02 13:28:07.164363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164167    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-token\" (UniqueName: \"kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-token\") pod \"f942a798-a6e7-427b-a16f-a6ed500bfa61\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") "
Nov 02 13:28:07.164363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164167    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-socket" pod "f942a798-a6e7-427b-a16f-a6ed500bfa61" (UID: "f942a798-a6e7-427b-a16f-a6ed500bfa61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:07.164363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164230    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-credential-socket" pod "f942a798-a6e7-427b-a16f-a6ed500bfa61" (UID: "f942a798-a6e7-427b-a16f-a6ed500bfa61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:07.164363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164243    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-kube-api-access-fw7xw\" (UniqueName: \"kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-kube-api-access-fw7xw\") pod \"f942a798-a6e7-427b-a16f-a6ed500bfa61\" (UID: \"f942a798-a6e7-427b-a16f-a6ed500bfa61\") "
Nov 02 13:28:07.164599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164526    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f942a798-a6e7-427b-a16f-a6ed500bfa61-istiod-ca-cert" pod "f942a798-a6e7-427b-a16f-a6ed500bfa61" (UID: "f942a798-a6e7-427b-a16f-a6ed500bfa61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:07.164865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164847    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:07.164947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164871    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:07.164947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164885    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f942a798-a6e7-427b-a16f-a6ed500bfa61-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:07.164947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164901    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:07.164947 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.164914    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:07.167255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.167228    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-podinfo" pod "f942a798-a6e7-427b-a16f-a6ed500bfa61" (UID: "f942a798-a6e7-427b-a16f-a6ed500bfa61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:28:07.167374 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.167266    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-token" pod "f942a798-a6e7-427b-a16f-a6ed500bfa61" (UID: "f942a798-a6e7-427b-a16f-a6ed500bfa61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:07.167374 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.167313    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-envoy" pod "f942a798-a6e7-427b-a16f-a6ed500bfa61" (UID: "f942a798-a6e7-427b-a16f-a6ed500bfa61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:28:07.167459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.167416    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-kube-api-access-fw7xw" pod "f942a798-a6e7-427b-a16f-a6ed500bfa61" (UID: "f942a798-a6e7-427b-a16f-a6ed500bfa61"). InnerVolumeSpecName "kube-api-access-fw7xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:07.266280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.266170    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:07.266280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.266209    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:07.266280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.266220    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:07.266280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.266233    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw7xw\" (UniqueName: \"kubernetes.io/projected/f942a798-a6e7-427b-a16f-a6ed500bfa61-kube-api-access-fw7xw\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:07.451215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.451184    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:07.905870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.905829    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8" event={"ID":"f942a798-a6e7-427b-a16f-a6ed500bfa61","Type":"ContainerDied","Data":"590d12ec1bff003763c18d774fb5cfbcec59747fd2ef16f67b82e3221544843e"}
Nov 02 13:28:07.905870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.905856    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"
Nov 02 13:28:07.906442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.905899    5879 scope.go:122] "RemoveContainer" containerID="d4ea8c0870cb2c9ee6524121127a3ba3eab870238c7f75bc51110f0abd1d4871"
Nov 02 13:28:07.948396 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:28:07.948359    5879 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf942a798_a6e7_427b_a16f_a6ed500bfa61.slice\": RecentStats: unable to find data in memory cache]"
Nov 02 13:28:07.978344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.978308    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"]
Nov 02 13:28:07.992604 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:07.992572    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8962e9-openshift-default-875c5bccf-7v7l8"]
Nov 02 13:28:08.018584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:08.018553    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" path="/var/lib/kubelet/pods/f942a798-a6e7-427b-a16f-a6ed500bfa61/volumes"
Nov 02 13:28:08.876895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:08.876857    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:09.142629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:09.142538    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:10.879229 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:10.879195    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:11.996414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:11.996358    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:12.207568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:12.207508    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:12.439198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:12.439130    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:12.859165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:12.859075    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:13.705202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:13.705165    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:13.718917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:13.718880    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:13.902593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:13.902555    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"]
Nov 02 13:28:13.938750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:13.938711    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"]
Nov 02 13:28:13.964157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:13.964077    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:28:14.075373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:14.075324    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:14.210193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:14.210159    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:14.729893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:14.729853    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:14.823400 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:28:14.823363    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:28:15.978923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:15.978892    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:16.645524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:16.645481    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:18.541392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:18.541357    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:18.903562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:18.903519    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg" podUID="038b3a32-6d30-4cd3-9941-8cda31f1b0ea" containerName="payload-processing" containerID="cri-o://13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5" gracePeriod=25
Nov 02 13:28:18.939503 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:18.939461    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2" podUID="4b4a3741-0289-4e75-b1bf-2881c27af2ce" containerName="payload-pre-processing" containerID="cri-o://de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc" gracePeriod=25
Nov 02 13:28:19.092785 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:28:19.092747    5879 logging.go:55] [core] [Channel #16734 SubChannel #16735]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.97:50052", ServerName: "10.134.1.97:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.97:50052: connect: connection refused"
Nov 02 13:28:19.266673 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.266644    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"
Nov 02 13:28:19.305823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.305796    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"
Nov 02 13:28:19.330794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.330764    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-kube-api-access-rpxvr\" (UniqueName: \"kubernetes.io/projected/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-kube-api-access-rpxvr\") pod \"038b3a32-6d30-4cd3-9941-8cda31f1b0ea\" (UID: \"038b3a32-6d30-4cd3-9941-8cda31f1b0ea\") "
Nov 02 13:28:19.330985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.330917    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-plugins-config-volume\") pod \"038b3a32-6d30-4cd3-9941-8cda31f1b0ea\" (UID: \"038b3a32-6d30-4cd3-9941-8cda31f1b0ea\") "
Nov 02 13:28:19.331357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.331333    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-plugins-config-volume" pod "038b3a32-6d30-4cd3-9941-8cda31f1b0ea" (UID: "038b3a32-6d30-4cd3-9941-8cda31f1b0ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:19.331502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.331487    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:19.333320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.333292    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-kube-api-access-rpxvr" pod "038b3a32-6d30-4cd3-9941-8cda31f1b0ea" (UID: "038b3a32-6d30-4cd3-9941-8cda31f1b0ea"). InnerVolumeSpecName "kube-api-access-rpxvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:19.373495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.373451    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:19.432698 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.432671    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4a3741-0289-4e75-b1bf-2881c27af2ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4a3741-0289-4e75-b1bf-2881c27af2ce-plugins-config-volume\") pod \"4b4a3741-0289-4e75-b1bf-2881c27af2ce\" (UID: \"4b4a3741-0289-4e75-b1bf-2881c27af2ce\") "
Nov 02 13:28:19.432904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.432890    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4a3741-0289-4e75-b1bf-2881c27af2ce-kube-api-access-tndtt\" (UniqueName: \"kubernetes.io/projected/4b4a3741-0289-4e75-b1bf-2881c27af2ce-kube-api-access-tndtt\") pod \"4b4a3741-0289-4e75-b1bf-2881c27af2ce\" (UID: \"4b4a3741-0289-4e75-b1bf-2881c27af2ce\") "
Nov 02 13:28:19.433174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.433141    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4a3741-0289-4e75-b1bf-2881c27af2ce-plugins-config-volume" pod "4b4a3741-0289-4e75-b1bf-2881c27af2ce" (UID: "4b4a3741-0289-4e75-b1bf-2881c27af2ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:19.433390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.433367    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpxvr\" (UniqueName: \"kubernetes.io/projected/038b3a32-6d30-4cd3-9941-8cda31f1b0ea-kube-api-access-rpxvr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:19.433494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.433396    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4a3741-0289-4e75-b1bf-2881c27af2ce-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:19.435547 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.435521    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4a3741-0289-4e75-b1bf-2881c27af2ce-kube-api-access-tndtt" pod "4b4a3741-0289-4e75-b1bf-2881c27af2ce" (UID: "4b4a3741-0289-4e75-b1bf-2881c27af2ce"). InnerVolumeSpecName "kube-api-access-tndtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:19.494681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.494586    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:19.534761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.534723    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tndtt\" (UniqueName: \"kubernetes.io/projected/4b4a3741-0289-4e75-b1bf-2881c27af2ce-kube-api-access-tndtt\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:19.856045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:19.855923    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:20.019248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.019214    5879 generic.go:356] "Generic (PLEG): container finished" podID="038b3a32-6d30-4cd3-9941-8cda31f1b0ea" containerID="13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5" exitCode=0
Nov 02 13:28:20.019447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.019352    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"
Nov 02 13:28:20.020944 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.020913    5879 generic.go:356] "Generic (PLEG): container finished" podID="4b4a3741-0289-4e75-b1bf-2881c27af2ce" containerID="de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc" exitCode=0
Nov 02 13:28:20.021174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.021158    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"
Nov 02 13:28:20.021814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.021787    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg" event={"ID":"038b3a32-6d30-4cd3-9941-8cda31f1b0ea","Type":"ContainerDied","Data":"13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5"}
Nov 02 13:28:20.021913 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.021852    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg" event={"ID":"038b3a32-6d30-4cd3-9941-8cda31f1b0ea","Type":"ContainerDied","Data":"99c638ff58810fdcfb22250b4f97be8308429ce2805a39bce16931d19629cb14"}
Nov 02 13:28:20.021913 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.021884    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2" event={"ID":"4b4a3741-0289-4e75-b1bf-2881c27af2ce","Type":"ContainerDied","Data":"de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc"}
Nov 02 13:28:20.021913 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.021910    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2" event={"ID":"4b4a3741-0289-4e75-b1bf-2881c27af2ce","Type":"ContainerDied","Data":"9ff606956ce22f356b07ddeff9c852a51ca468b7578a5df9156c571be3b034f3"}
Nov 02 13:28:20.022152 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.021932    5879 scope.go:122] "RemoveContainer" containerID="13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5"
Nov 02 13:28:20.036546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.036518    5879 scope.go:122] "RemoveContainer" containerID="13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5"
Nov 02 13:28:20.037092 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:28:20.037055    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5\": container with ID starting with 13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5 not found: ID does not exist" containerID="13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5"
Nov 02 13:28:20.037200 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.037106    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5"} err="failed to get container status \"13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5\": rpc error: code = NotFound desc = could not find container \"13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5\": container with ID starting with 13c6c9e3a6e35c6e4790ac2e54d64b2986fa2351085ca094d17776047b4417a5 not found: ID does not exist"
Nov 02 13:28:20.037200 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.037132    5879 scope.go:122] "RemoveContainer" containerID="de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc"
Nov 02 13:28:20.049412 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.049390    5879 scope.go:122] "RemoveContainer" containerID="de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc"
Nov 02 13:28:20.049755 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:28:20.049733    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc\": container with ID starting with de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc not found: ID does not exist" containerID="de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc"
Nov 02 13:28:20.049836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.049764    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc"} err="failed to get container status \"de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc\": rpc error: code = NotFound desc = could not find container \"de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc\": container with ID starting with de74ac2a9eeb244e5fe75ffe6cd37c041a187d699040721461bcccf06019a0cc not found: ID does not exist"
Nov 02 13:28:20.078387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.078353    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"]
Nov 02 13:28:20.092305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.092267    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b8a8dc4c-84fd646865-m5zmg"]
Nov 02 13:28:20.092483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.092389    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2" podUID="4b4a3741-0289-4e75-b1bf-2881c27af2ce" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.97:50052\" within 1s: context deadline exceeded"
Nov 02 13:28:20.139317 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.139284    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"]
Nov 02 13:28:20.143143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.143118    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b8a8dc4c-58df56c5d7-xw4q2"]
Nov 02 13:28:20.383399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:20.383341    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:22.020449 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:22.020410    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038b3a32-6d30-4cd3-9941-8cda31f1b0ea" path="/var/lib/kubelet/pods/038b3a32-6d30-4cd3-9941-8cda31f1b0ea/volumes"
Nov 02 13:28:22.020967 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:22.020872    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4a3741-0289-4e75-b1bf-2881c27af2ce" path="/var/lib/kubelet/pods/4b4a3741-0289-4e75-b1bf-2881c27af2ce/volumes"
Nov 02 13:28:22.079429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:22.079394    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:22.396525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:22.396495    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:24.055232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:24.055194    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:25.091491 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.091451    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"]
Nov 02 13:28:25.150112 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.150028    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"]
Nov 02 13:28:25.151015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.150954    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy"
Nov 02 13:28:25.151015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.150984    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy"
Nov 02 13:28:25.151015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.151021    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4b4a3741-0289-4e75-b1bf-2881c27af2ce" containerName="payload-pre-processing"
Nov 02 13:28:25.151280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.151029    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4a3741-0289-4e75-b1bf-2881c27af2ce" containerName="payload-pre-processing"
Nov 02 13:28:25.151280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.151092    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy"
Nov 02 13:28:25.151280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.151102    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy"
Nov 02 13:28:25.151280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.151142    5879 cpu_manager.go:414] "containerMap: removing container" podUID="038b3a32-6d30-4cd3-9941-8cda31f1b0ea" containerName="payload-processing"
Nov 02 13:28:25.151280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.151149    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038b3a32-6d30-4cd3-9941-8cda31f1b0ea" containerName="payload-processing"
Nov 02 13:28:25.151570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.151358    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy"
Nov 02 13:28:25.151570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.151374    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="038b3a32-6d30-4cd3-9941-8cda31f1b0ea" containerName="payload-processing"
Nov 02 13:28:25.151570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.151388    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f942a798-a6e7-427b-a16f-a6ed500bfa61" containerName="istio-proxy"
Nov 02 13:28:25.151570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.151398    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4a3741-0289-4e75-b1bf-2881c27af2ce" containerName="payload-pre-processing"
Nov 02 13:28:25.157341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.157315    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.165039 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.164986    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"]
Nov 02 13:28:25.179292 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.179246    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-43be82\"/\"openshift-service-ca.crt\""
Nov 02 13:28:25.179509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.179344    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-43be82\"/\"rate-test-model-43be82-kserve-self-signed-certs\""
Nov 02 13:28:25.179509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.179348    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-43be82\"/\"kube-root-ca.crt\""
Nov 02 13:28:25.179509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.179381    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-43be82\"/\"default-dockercfg-w5zw7\""
Nov 02 13:28:25.184431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.184406    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"]
Nov 02 13:28:25.228394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.228307    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:25.327462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.327422    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712a34b5-6d02-4f7b-9f07-ae896cacec19-tls-certs\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.327688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.327505    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-home\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.327688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.327602    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-model-cache\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.327822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.327752    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p496x\" (UniqueName: \"kubernetes.io/projected/712a34b5-6d02-4f7b-9f07-ae896cacec19-kube-api-access-p496x\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.327822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.327796    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-tmp-dir\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.327939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.327840    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-dshm\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.354460 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.354368    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:25.429788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.429748    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712a34b5-6d02-4f7b-9f07-ae896cacec19-tls-certs\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.430032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.429813    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-home\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.430032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.429847    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-model-cache\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.430032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.429946    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p496x\" (UniqueName: \"kubernetes.io/projected/712a34b5-6d02-4f7b-9f07-ae896cacec19-kube-api-access-p496x\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.430032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.429988    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-tmp-dir\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.430280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.430082    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-dshm\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.430434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.430407    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-home\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.430502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.430485    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-tmp-dir\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.430874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.430763    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-model-cache\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.432547 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.432518    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-dshm\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.432862 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.432840    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712a34b5-6d02-4f7b-9f07-ae896cacec19-tls-certs\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.456399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.456357    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p496x\" (UniqueName: \"kubernetes.io/projected/712a34b5-6d02-4f7b-9f07-ae896cacec19-kube-api-access-p496x\") pod \"rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") " pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.474308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.474267    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:25.863302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:25.863274    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"]
Nov 02 13:28:26.063275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:26.063238    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:26.070200 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:26.070161    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb" event={"ID":"712a34b5-6d02-4f7b-9f07-ae896cacec19","Type":"ContainerStarted","Data":"90072cbd0032a5112d618e94191ae28e815bb71307c87e131c83a74269b207e6"}
Nov 02 13:28:26.070355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:26.070209    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb" event={"ID":"712a34b5-6d02-4f7b-9f07-ae896cacec19","Type":"ContainerStarted","Data":"a4fee678f2a2168eed9538512afdd6af23bf8d172ce457131837133cc8f059c7"}
Nov 02 13:28:26.116401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:26.116337    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb" podStartSLOduration=1.116313662 podStartE2EDuration="1.116313662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:28:26.115105821 +0000 UTC m=+3652.221808675" watchObservedRunningTime="2026-11-02 13:28:26.116313662 +0000 UTC m=+3652.223016520"
Nov 02 13:28:26.475191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:26.475097    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:27.191269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.191232    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"]
Nov 02 13:28:27.197247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.197197    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.224506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.220262    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"]
Nov 02 13:28:27.224506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.221625    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-ed56b9\"/\"openshift-service-ca.crt\""
Nov 02 13:28:27.226572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.226544    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-ed56b9\"/\"kube-root-ca.crt\""
Nov 02 13:28:27.226716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.226674    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-ed56b9\"/\"default-dockercfg-fvspn\""
Nov 02 13:28:27.228780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.228746    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-ed56b9\"/\"sub-test-model-ed56b9-kserve-self-signed-certs\""
Nov 02 13:28:27.280207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.280172    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:27.382611 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.382568    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tls-certs\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.383245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.382945    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-model-cache\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.383245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.383051    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-home\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.383245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.383228    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tmp-dir\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.383245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.383265    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn8gm\" (UniqueName: \"kubernetes.io/projected/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-kube-api-access-nn8gm\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.383603 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.383502    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-dshm\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.485208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.485099    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tmp-dir\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.485208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.485162    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn8gm\" (UniqueName: \"kubernetes.io/projected/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-kube-api-access-nn8gm\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.485495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.485290    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-dshm\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.485495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.485406    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tls-certs\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.485495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.485455    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-model-cache\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.485495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.485488    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-home\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.485730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.485626    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tmp-dir\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.485965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.485934    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-home\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.486123 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.485969    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-model-cache\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.488685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.488656    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tls-certs\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.488969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.488938    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-dshm\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.512512 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.512478    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn8gm\" (UniqueName: \"kubernetes.io/projected/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-kube-api-access-nn8gm\") pod \"sub-test-model-ed56b9-kserve-666c86c895-r4vzm\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") " pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.525568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.525525    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:27.604575 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.604077    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:27.843982 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:28:27.843943    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc53b600d_86f8_4b1e_84c7_0f3a33b3d53f.slice/crio-60da59e159712196b9a9c5b0c04e42bcf28e90eb309f85b93d8024e85c9af25c WatchSource:0}: Error finding container 60da59e159712196b9a9c5b0c04e42bcf28e90eb309f85b93d8024e85c9af25c: Status 404 returned error can't find the container with id 60da59e159712196b9a9c5b0c04e42bcf28e90eb309f85b93d8024e85c9af25c
Nov 02 13:28:27.845276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.845243    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"]
Nov 02 13:28:27.962183 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:27.962145    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:28.089372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:28.089318    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm" event={"ID":"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f","Type":"ContainerStarted","Data":"aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a"}
Nov 02 13:28:28.089372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:28.089372    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm" event={"ID":"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f","Type":"ContainerStarted","Data":"60da59e159712196b9a9c5b0c04e42bcf28e90eb309f85b93d8024e85c9af25c"}
Nov 02 13:28:28.135256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:28.135202    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm" podStartSLOduration=1.135182124 podStartE2EDuration="1.135182124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:28:28.134215415 +0000 UTC m=+3654.240918267" watchObservedRunningTime="2026-11-02 13:28:28.135182124 +0000 UTC m=+3654.241884977"
Nov 02 13:28:28.526522 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:28.526441    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:29.867521 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:29.867480    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:30.092666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:30.092608    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7" podUID="164d08f2-4367-4d53-a393-1995d9a4d12e" containerName="payload-processing" containerID="cri-o://1bb2c6ddae673aaa209857cc7a3fe45b00234dd40e166c178f894fb8b77bbfa6" gracePeriod=25
Nov 02 13:28:30.165938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:30.165847    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw" podUID="3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2" containerName="payload-pre-processing" containerID="cri-o://e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec" gracePeriod=25
Nov 02 13:28:30.479588 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:30.479504    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:31.539787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:31.539736    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:33.305421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:33.305378    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7" podUID="164d08f2-4367-4d53-a393-1995d9a4d12e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:28:33.305905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:33.305781    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw" podUID="3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:28:33.383596 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:33.383559    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:33.674725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:33.674692    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:34.037042 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:34.036945    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"]
Nov 02 13:28:34.050277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:34.050248    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d25d2853-xplwk"]
Nov 02 13:28:35.162707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.162674    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:35.475341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.475249    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:35.488592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.488563    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:35.615746 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.615704    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"]
Nov 02 13:28:35.620618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.620600    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.640743 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.640711    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"]
Nov 02 13:28:35.644356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.644327    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-1aa18c\"/\"rate-test-model-1aa18c-kserve-self-signed-certs\""
Nov 02 13:28:35.644356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.644352    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-1aa18c\"/\"openshift-service-ca.crt\""
Nov 02 13:28:35.644356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.644363    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-1aa18c\"/\"kube-root-ca.crt\""
Nov 02 13:28:35.697562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.697519    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-model-cache\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.697749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.697696    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sffc6\" (UniqueName: \"kubernetes.io/projected/a5ae966c-6997-446c-a6b6-52393972b61c-kube-api-access-sffc6\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.697950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.697930    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-dshm\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.697986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.697968    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ae966c-6997-446c-a6b6-52393972b61c-tls-certs\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.698068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.698016    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-home\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.698128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.698108    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-tmp-dir\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.698978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.698953    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:35.711816 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.711792    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-1aa18c\"/\"default-dockercfg-2xjds\""
Nov 02 13:28:35.800263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.800160    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-model-cache\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.800263 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.800216    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffc6\" (UniqueName: \"kubernetes.io/projected/a5ae966c-6997-446c-a6b6-52393972b61c-kube-api-access-sffc6\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.800528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.800433    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-dshm\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.800528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.800465    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ae966c-6997-446c-a6b6-52393972b61c-tls-certs\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.800528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.800494    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-home\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.800703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.800562    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-tmp-dir\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.800703 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.800654    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-model-cache\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.800900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.800873    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-home\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.800981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.800967    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-tmp-dir\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.803032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.802977    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-dshm\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.803232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.803216    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ae966c-6997-446c-a6b6-52393972b61c-tls-certs\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.828024 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.827974    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffc6\" (UniqueName: \"kubernetes.io/projected/a5ae966c-6997-446c-a6b6-52393972b61c-kube-api-access-sffc6\") pod \"rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") " pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:35.937675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:35.937632    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:36.021778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:36.021654    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05cd217b-d890-4bdf-a282-5deb6decb94d" path="/var/lib/kubelet/pods/05cd217b-d890-4bdf-a282-5deb6decb94d/volumes"
Nov 02 13:28:36.162336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:36.162301    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:28:36.267871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:36.267842    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"]
Nov 02 13:28:36.268386 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:28:36.268362    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5ae966c_6997_446c_a6b6_52393972b61c.slice/crio-6961ec147ce7ca98fd866b2f9e49c2301935f864e837e1950faefd3db4da1922 WatchSource:0}: Error finding container 6961ec147ce7ca98fd866b2f9e49c2301935f864e837e1950faefd3db4da1922: Status 404 returned error can't find the container with id 6961ec147ce7ca98fd866b2f9e49c2301935f864e837e1950faefd3db4da1922
Nov 02 13:28:36.940739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:36.940703    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:37.160478 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:37.160436    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt" event={"ID":"a5ae966c-6997-446c-a6b6-52393972b61c","Type":"ContainerStarted","Data":"61df33f8c78f45658bb0f858fc3360b04eb74f6d53f6411b9487c4b679725daa"}
Nov 02 13:28:37.160698 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:37.160492    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt" event={"ID":"a5ae966c-6997-446c-a6b6-52393972b61c","Type":"ContainerStarted","Data":"6961ec147ce7ca98fd866b2f9e49c2301935f864e837e1950faefd3db4da1922"}
Nov 02 13:28:37.207174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:37.207062    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt" podStartSLOduration=2.207044991 podStartE2EDuration="2.207044991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:28:37.205422178 +0000 UTC m=+3663.312125041" watchObservedRunningTime="2026-11-02 13:28:37.207044991 +0000 UTC m=+3663.313747841"
Nov 02 13:28:37.526425 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:37.526327    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:37.539749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:37.539715    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:37.938824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:37.938790    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:37.957097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:37.957062    5879 scope.go:122] "RemoveContainer" containerID="8df4d9430d7948938fb57ddcb47a2ada5d38492d941266e25b536a4906ef6e18"
Nov 02 13:28:38.180192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:38.180164    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:28:38.666762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:38.666723    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:40.071343 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:40.071309    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:40.760677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:40.760645    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:41.020465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:41.020380    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:42.693452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:42.693417    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:43.305689 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:43.305649    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7" podUID="164d08f2-4367-4d53-a393-1995d9a4d12e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:28:43.305889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:43.305826    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw" podUID="3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:28:44.432682 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:44.432647    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:44.821984 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:28:44.821896    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:28:45.937806 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:45.937765    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:45.954138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:45.954103    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:46.162538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:46.162496    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:46.257605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:46.257530    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:28:47.977287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:47.977242    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:49.875111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:49.875074    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:50.002617 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.002585    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:50.284569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.284530    5879 generic.go:356] "Generic (PLEG): container finished" podID="164d08f2-4367-4d53-a393-1995d9a4d12e" containerID="1bb2c6ddae673aaa209857cc7a3fe45b00234dd40e166c178f894fb8b77bbfa6" exitCode=0
Nov 02 13:28:50.284726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.284619    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7" event={"ID":"164d08f2-4367-4d53-a393-1995d9a4d12e","Type":"ContainerDied","Data":"1bb2c6ddae673aaa209857cc7a3fe45b00234dd40e166c178f894fb8b77bbfa6"}
Nov 02 13:28:50.489196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.489169    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"
Nov 02 13:28:50.493720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.493693    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"
Nov 02 13:28:50.617575 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.617540    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-plugins-config-volume\") pod \"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2\" (UID: \"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2\") "
Nov 02 13:28:50.617804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.617790    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164d08f2-4367-4d53-a393-1995d9a4d12e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164d08f2-4367-4d53-a393-1995d9a4d12e-plugins-config-volume\") pod \"164d08f2-4367-4d53-a393-1995d9a4d12e\" (UID: \"164d08f2-4367-4d53-a393-1995d9a4d12e\") "
Nov 02 13:28:50.617904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.617847    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164d08f2-4367-4d53-a393-1995d9a4d12e-kube-api-access-vwk92\" (UniqueName: \"kubernetes.io/projected/164d08f2-4367-4d53-a393-1995d9a4d12e-kube-api-access-vwk92\") pod \"164d08f2-4367-4d53-a393-1995d9a4d12e\" (UID: \"164d08f2-4367-4d53-a393-1995d9a4d12e\") "
Nov 02 13:28:50.617979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.617959    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-kube-api-access-mp4vp\" (UniqueName: \"kubernetes.io/projected/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-kube-api-access-mp4vp\") pod \"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2\" (UID: \"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2\") "
Nov 02 13:28:50.618640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.618242    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164d08f2-4367-4d53-a393-1995d9a4d12e-plugins-config-volume" pod "164d08f2-4367-4d53-a393-1995d9a4d12e" (UID: "164d08f2-4367-4d53-a393-1995d9a4d12e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:50.618640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.618601    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164d08f2-4367-4d53-a393-1995d9a4d12e-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:50.619322 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.619294    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-plugins-config-volume" pod "3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2" (UID: "3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:28:50.621731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.621685    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164d08f2-4367-4d53-a393-1995d9a4d12e-kube-api-access-vwk92" pod "164d08f2-4367-4d53-a393-1995d9a4d12e" (UID: "164d08f2-4367-4d53-a393-1995d9a4d12e"). InnerVolumeSpecName "kube-api-access-vwk92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:50.621847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.621809    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-kube-api-access-mp4vp" pod "3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2" (UID: "3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2"). InnerVolumeSpecName "kube-api-access-mp4vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:28:50.668312 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.668277    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:50.720377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.720292    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwk92\" (UniqueName: \"kubernetes.io/projected/164d08f2-4367-4d53-a393-1995d9a4d12e-kube-api-access-vwk92\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:50.720377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.720327    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4vp\" (UniqueName: \"kubernetes.io/projected/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-kube-api-access-mp4vp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:50.720377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:50.720344    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:28:51.126545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.126510    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:51.294832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.294799    5879 generic.go:356] "Generic (PLEG): container finished" podID="3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2" containerID="e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec" exitCode=0
Nov 02 13:28:51.295070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.294853    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"
Nov 02 13:28:51.295070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.294884    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw" event={"ID":"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2","Type":"ContainerDied","Data":"e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec"}
Nov 02 13:28:51.295070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.294935    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw" event={"ID":"3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2","Type":"ContainerDied","Data":"0451713003ee86f06885bbba16c389939c50a829826d1d795348247bef376da8"}
Nov 02 13:28:51.295070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.294961    5879 scope.go:122] "RemoveContainer" containerID="e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec"
Nov 02 13:28:51.296812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.296790    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7" event={"ID":"164d08f2-4367-4d53-a393-1995d9a4d12e","Type":"ContainerDied","Data":"64337e1ac6dcc5f0cf3d14f082987c3e42eba5f1a60427a779a9101d3fb5ebde"}
Nov 02 13:28:51.296933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.296907    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"
Nov 02 13:28:51.309176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.309157    5879 scope.go:122] "RemoveContainer" containerID="e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec"
Nov 02 13:28:51.309473 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:28:51.309454    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec\": container with ID starting with e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec not found: ID does not exist" containerID="e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec"
Nov 02 13:28:51.309664 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.309481    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec"} err="failed to get container status \"e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec\": rpc error: code = NotFound desc = could not find container \"e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec\": container with ID starting with e8374fb479be8c3d41d2cad17226ea1aa3f396123f794fb0ac4c18a387db43ec not found: ID does not exist"
Nov 02 13:28:51.309664 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.309498    5879 scope.go:122] "RemoveContainer" containerID="1bb2c6ddae673aaa209857cc7a3fe45b00234dd40e166c178f894fb8b77bbfa6"
Nov 02 13:28:51.358009 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.357960    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"]
Nov 02 13:28:51.363729 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.363698    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ef5efe-5c765978c-k7pd7"]
Nov 02 13:28:51.400027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.399878    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"]
Nov 02 13:28:51.407230 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.407199    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ef5efe-6cb8cb4479-49thw"]
Nov 02 13:28:51.827908 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.827825    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:51.955727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:51.955694    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:52.018725 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:52.018693    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164d08f2-4367-4d53-a393-1995d9a4d12e" path="/var/lib/kubelet/pods/164d08f2-4367-4d53-a393-1995d9a4d12e/volumes"
Nov 02 13:28:52.019242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:52.019222    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2" path="/var/lib/kubelet/pods/3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2/volumes"
Nov 02 13:28:53.502310 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:53.502261    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:54.180132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:54.180097    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:56.022665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:56.022624    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:56.944116 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:56.944085    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:57.092856 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:57.092816    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:57.749132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:57.749098    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:57.900252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:57.900209    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:57.979715 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:57.979672    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"]
Nov 02 13:28:58.034896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:58.034802    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"]
Nov 02 13:28:58.184020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:58.183969    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:58.311814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:58.311723    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:58.646364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:58.646316    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:58.776366 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:58.776309    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:58.978891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:58.978785    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:59.143323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:59.143289    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:28:59.944407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:28:59.944371    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:00.452194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.452158    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"]
Nov 02 13:29:00.453528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.453493    5879 cpu_manager.go:414] "containerMap: removing container" podUID="164d08f2-4367-4d53-a393-1995d9a4d12e" containerName="payload-processing"
Nov 02 13:29:00.453528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.453527    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164d08f2-4367-4d53-a393-1995d9a4d12e" containerName="payload-processing"
Nov 02 13:29:00.453707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.453602    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2" containerName="payload-pre-processing"
Nov 02 13:29:00.453707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.453611    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2" containerName="payload-pre-processing"
Nov 02 13:29:00.453830 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.453816    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="164d08f2-4367-4d53-a393-1995d9a4d12e" containerName="payload-processing"
Nov 02 13:29:00.453868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.453846    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed43f8f-48f7-46a7-8fc9-f96f0a13d1e2" containerName="payload-pre-processing"
Nov 02 13:29:00.459933 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.459908    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.477530 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.477497    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"]
Nov 02 13:29:00.489028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.487969    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-1e14d4\"/\"sub-test-model-1e14d4-kserve-self-signed-certs\""
Nov 02 13:29:00.489028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.487973    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-1e14d4\"/\"default-dockercfg-6n4mn\""
Nov 02 13:29:00.490348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.490312    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-1e14d4\"/\"kube-root-ca.crt\""
Nov 02 13:29:00.490348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.490315    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-1e14d4\"/\"openshift-service-ca.crt\""
Nov 02 13:29:00.531372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.531341    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:00.593160 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.593118    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tls-certs\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.593360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.593244    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-dshm\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.593360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.593307    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rshr\" (UniqueName: \"kubernetes.io/projected/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-kube-api-access-4rshr\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.593360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.593342    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tmp-dir\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.593627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.593508    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-model-cache\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.593627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.593603    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-home\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.701744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.699645    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tls-certs\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.701744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.699731    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-dshm\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.701744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.699789    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rshr\" (UniqueName: \"kubernetes.io/projected/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-kube-api-access-4rshr\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.701744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.699823    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tmp-dir\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.701744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.699875    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-model-cache\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.701744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.699946    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-home\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.701744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.700510    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-home\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.702568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.701815    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tmp-dir\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.702568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.702095    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-model-cache\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.707401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.707370    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-dshm\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.708521 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.708486    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tls-certs\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.729197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.729161    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rshr\" (UniqueName: \"kubernetes.io/projected/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-kube-api-access-4rshr\") pod \"sub-test-model-1e14d4-kserve-6cd8797bf9-whffv\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") " pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:00.775092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:00.775036    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:01.098073 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:29:01.098037    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc97dbd0f_2281_49dc_8ccf_a9c47da1f9da.slice/crio-2a9c363a636839dc459bb8f2fd7ca2e41fe090c87ea265f95f788e8eb1deba0b WatchSource:0}: Error finding container 2a9c363a636839dc459bb8f2fd7ca2e41fe090c87ea265f95f788e8eb1deba0b: Status 404 returned error can't find the container with id 2a9c363a636839dc459bb8f2fd7ca2e41fe090c87ea265f95f788e8eb1deba0b
Nov 02 13:29:01.099974 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:01.099947    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"]
Nov 02 13:29:01.408306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:01.408265    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv" event={"ID":"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da","Type":"ContainerStarted","Data":"cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0"}
Nov 02 13:29:01.408514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:01.408315    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv" event={"ID":"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da","Type":"ContainerStarted","Data":"2a9c363a636839dc459bb8f2fd7ca2e41fe090c87ea265f95f788e8eb1deba0b"}
Nov 02 13:29:01.453648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:01.453586    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv" podStartSLOduration=1.45355996 podStartE2EDuration="1.45355996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:29:01.452419751 +0000 UTC m=+3687.559122602" watchObservedRunningTime="2026-11-02 13:29:01.45355996 +0000 UTC m=+3687.560262810"
Nov 02 13:29:01.667722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:01.667610    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:01.776080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:01.776047    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:02.979971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:02.979929    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx" podUID="2117da57-e909-43d4-a471-c0f4b3c0f13d" containerName="payload-processing" containerID="cri-o://a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341" gracePeriod=25
Nov 02 13:29:03.035306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:03.035263    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq" podUID="111b30d0-5386-4fc5-bc65-6108173dede5" containerName="payload-pre-processing" containerID="cri-o://3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca" gracePeriod=25
Nov 02 13:29:03.478619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:03.478565    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:05.004464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:05.004419    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx" podUID="2117da57-e909-43d4-a471-c0f4b3c0f13d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:29:05.004894 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:05.004428    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq" podUID="111b30d0-5386-4fc5-bc65-6108173dede5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:29:05.200728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:05.200682    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:05.343446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:05.343362    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:05.495231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:05.495196    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:05.750329 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:05.750294    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:06.752065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:06.752010    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:07.639281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:07.639247    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:09.421780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:09.421744    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:10.750239 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:10.750193    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"]
Nov 02 13:29:10.775565 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:10.775533    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:10.785937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:10.785901    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"]
Nov 02 13:29:10.789851 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:10.789828    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:10.926296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:10.926262    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:11.050942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:11.050847    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:11.495504 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:11.495476    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:29:11.730896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:11.730858    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:12.260515 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:12.260482    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:13.553868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:13.553793    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:13.779042 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:13.778983    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:14.822265 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:29:14.822230    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:29:15.004618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:15.004578    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq" podUID="111b30d0-5386-4fc5-bc65-6108173dede5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:29:15.004799 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:15.004620    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx" podUID="2117da57-e909-43d4-a471-c0f4b3c0f13d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:29:15.487855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:15.487820    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:15.751101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:15.751007    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg" podUID="97ae8f04-3aba-4de2-9987-4f4bc895e558" containerName="payload-processing" containerID="cri-o://72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1" gracePeriod=25
Nov 02 13:29:15.786717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:15.786673    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm" podUID="14e890f7-843c-478e-97d5-4f343aaa9ad4" containerName="payload-pre-processing" containerID="cri-o://44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a" gracePeriod=25
Nov 02 13:29:16.396088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:16.396037    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg" podUID="97ae8f04-3aba-4de2-9987-4f4bc895e558" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:29:16.396088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:16.396077    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm" podUID="14e890f7-843c-478e-97d5-4f343aaa9ad4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:29:17.295601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:17.295566    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:19.052867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:19.052829    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:20.272167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:20.272134    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:20.409824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:20.409791    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:20.829738 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:20.829700    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:21.081532 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:21.081436    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:21.538337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:21.538303    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:22.400979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:22.400938    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:22.852806 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:22.852715    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:23.339044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.339018    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"
Nov 02 13:29:23.361773 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.361743    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2117da57-e909-43d4-a471-c0f4b3c0f13d-kube-api-access-6bvk5\" (UniqueName: \"kubernetes.io/projected/2117da57-e909-43d4-a471-c0f4b3c0f13d-kube-api-access-6bvk5\") pod \"2117da57-e909-43d4-a471-c0f4b3c0f13d\" (UID: \"2117da57-e909-43d4-a471-c0f4b3c0f13d\") "
Nov 02 13:29:23.361899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.361879    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2117da57-e909-43d4-a471-c0f4b3c0f13d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2117da57-e909-43d4-a471-c0f4b3c0f13d-plugins-config-volume\") pod \"2117da57-e909-43d4-a471-c0f4b3c0f13d\" (UID: \"2117da57-e909-43d4-a471-c0f4b3c0f13d\") "
Nov 02 13:29:23.362358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.362306    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2117da57-e909-43d4-a471-c0f4b3c0f13d-plugins-config-volume" pod "2117da57-e909-43d4-a471-c0f4b3c0f13d" (UID: "2117da57-e909-43d4-a471-c0f4b3c0f13d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:23.362505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.362457    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2117da57-e909-43d4-a471-c0f4b3c0f13d-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:23.364143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.364121    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2117da57-e909-43d4-a471-c0f4b3c0f13d-kube-api-access-6bvk5" pod "2117da57-e909-43d4-a471-c0f4b3c0f13d" (UID: "2117da57-e909-43d4-a471-c0f4b3c0f13d"). InnerVolumeSpecName "kube-api-access-6bvk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:23.374367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.374338    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"
Nov 02 13:29:23.463532 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.463431    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111b30d0-5386-4fc5-bc65-6108173dede5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111b30d0-5386-4fc5-bc65-6108173dede5-plugins-config-volume\") pod \"111b30d0-5386-4fc5-bc65-6108173dede5\" (UID: \"111b30d0-5386-4fc5-bc65-6108173dede5\") "
Nov 02 13:29:23.463727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.463685    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111b30d0-5386-4fc5-bc65-6108173dede5-kube-api-access-6r6pb\" (UniqueName: \"kubernetes.io/projected/111b30d0-5386-4fc5-bc65-6108173dede5-kube-api-access-6r6pb\") pod \"111b30d0-5386-4fc5-bc65-6108173dede5\" (UID: \"111b30d0-5386-4fc5-bc65-6108173dede5\") "
Nov 02 13:29:23.463918 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.463892    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111b30d0-5386-4fc5-bc65-6108173dede5-plugins-config-volume" pod "111b30d0-5386-4fc5-bc65-6108173dede5" (UID: "111b30d0-5386-4fc5-bc65-6108173dede5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:23.464474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.464446    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111b30d0-5386-4fc5-bc65-6108173dede5-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:23.464474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.464475    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvk5\" (UniqueName: \"kubernetes.io/projected/2117da57-e909-43d4-a471-c0f4b3c0f13d-kube-api-access-6bvk5\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:23.466277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.466256    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111b30d0-5386-4fc5-bc65-6108173dede5-kube-api-access-6r6pb" pod "111b30d0-5386-4fc5-bc65-6108173dede5" (UID: "111b30d0-5386-4fc5-bc65-6108173dede5"). InnerVolumeSpecName "kube-api-access-6r6pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:23.566348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.566308    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r6pb\" (UniqueName: \"kubernetes.io/projected/111b30d0-5386-4fc5-bc65-6108173dede5-kube-api-access-6r6pb\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:23.597598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.597562    5879 generic.go:356] "Generic (PLEG): container finished" podID="2117da57-e909-43d4-a471-c0f4b3c0f13d" containerID="a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341" exitCode=0
Nov 02 13:29:23.597822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.597622    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"
Nov 02 13:29:23.597822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.597643    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx" event={"ID":"2117da57-e909-43d4-a471-c0f4b3c0f13d","Type":"ContainerDied","Data":"a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341"}
Nov 02 13:29:23.597822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.597723    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx" event={"ID":"2117da57-e909-43d4-a471-c0f4b3c0f13d","Type":"ContainerDied","Data":"3e69eb925a7901b39553ca555cda37124f7313292f199c6eb1d7554546c96da6"}
Nov 02 13:29:23.597822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.597748    5879 scope.go:122] "RemoveContainer" containerID="a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341"
Nov 02 13:29:23.599409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.599384    5879 generic.go:356] "Generic (PLEG): container finished" podID="111b30d0-5386-4fc5-bc65-6108173dede5" containerID="3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca" exitCode=0
Nov 02 13:29:23.599519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.599483    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq" event={"ID":"111b30d0-5386-4fc5-bc65-6108173dede5","Type":"ContainerDied","Data":"3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca"}
Nov 02 13:29:23.599568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.599545    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq" event={"ID":"111b30d0-5386-4fc5-bc65-6108173dede5","Type":"ContainerDied","Data":"9487f0e2f89bc6c41603a2517fc26f6b4157ef4dd80d32b185d11f049f59d3bf"}
Nov 02 13:29:23.599646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.599630    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"
Nov 02 13:29:23.616864 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.616843    5879 scope.go:122] "RemoveContainer" containerID="a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341"
Nov 02 13:29:23.617211 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:29:23.617192    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341\": container with ID starting with a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341 not found: ID does not exist" containerID="a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341"
Nov 02 13:29:23.617271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.617220    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341"} err="failed to get container status \"a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341\": rpc error: code = NotFound desc = could not find container \"a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341\": container with ID starting with a6808cf5c766a621be46b33581aa25f7ff6ff574d5950f953ec70274f29c9341 not found: ID does not exist"
Nov 02 13:29:23.617271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.617236    5879 scope.go:122] "RemoveContainer" containerID="3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca"
Nov 02 13:29:23.627779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.627757    5879 scope.go:122] "RemoveContainer" containerID="3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca"
Nov 02 13:29:23.628145 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:29:23.628124    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca\": container with ID starting with 3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca not found: ID does not exist" containerID="3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca"
Nov 02 13:29:23.628257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.628152    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca"} err="failed to get container status \"3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca\": rpc error: code = NotFound desc = could not find container \"3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca\": container with ID starting with 3dddc9fca9a06f14f6ad61a6e883082c0c58cd15fd90471e6ec2b753c9ec2cca not found: ID does not exist"
Nov 02 13:29:23.655476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.655443    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"]
Nov 02 13:29:23.658630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.658599    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-936529-f869b5f7f-5rhnq"]
Nov 02 13:29:23.696796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.696753    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"]
Nov 02 13:29:23.704267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:23.704235    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-936529-5c8d858c85-p6stx"]
Nov 02 13:29:24.018623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:24.018590    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111b30d0-5386-4fc5-bc65-6108173dede5" path="/var/lib/kubelet/pods/111b30d0-5386-4fc5-bc65-6108173dede5/volumes"
Nov 02 13:29:24.019151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:24.019132    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2117da57-e909-43d4-a471-c0f4b3c0f13d" path="/var/lib/kubelet/pods/2117da57-e909-43d4-a471-c0f4b3c0f13d/volumes"
Nov 02 13:29:24.830211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:24.830171    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:26.395940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:26.395906    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg" podUID="97ae8f04-3aba-4de2-9987-4f4bc895e558" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:29:26.395940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:26.395923    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm" podUID="14e890f7-843c-478e-97d5-4f343aaa9ad4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:29:26.796285 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:26.796156    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:28.762151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:28.762116    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:30.868728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:30.868688    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:31.120399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:31.120322    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:31.467134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:31.467046    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"]
Nov 02 13:29:31.523708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:31.523675    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"]
Nov 02 13:29:31.724144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:31.724053    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:31.926121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:31.926081    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:33.305513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:33.305475    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:34.854494 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:34.854457    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:35.598384 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:35.598346    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:36.091557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.091522    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"
Nov 02 13:29:36.118340 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.118314    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"
Nov 02 13:29:36.257230 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.257201    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ae8f04-3aba-4de2-9987-4f4bc895e558-kube-api-access-bj4bw\" (UniqueName: \"kubernetes.io/projected/97ae8f04-3aba-4de2-9987-4f4bc895e558-kube-api-access-bj4bw\") pod \"97ae8f04-3aba-4de2-9987-4f4bc895e558\" (UID: \"97ae8f04-3aba-4de2-9987-4f4bc895e558\") "
Nov 02 13:29:36.257440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.257401    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e890f7-843c-478e-97d5-4f343aaa9ad4-kube-api-access-vqd8r\" (UniqueName: \"kubernetes.io/projected/14e890f7-843c-478e-97d5-4f343aaa9ad4-kube-api-access-vqd8r\") pod \"14e890f7-843c-478e-97d5-4f343aaa9ad4\" (UID: \"14e890f7-843c-478e-97d5-4f343aaa9ad4\") "
Nov 02 13:29:36.257541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.257525    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e890f7-843c-478e-97d5-4f343aaa9ad4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e890f7-843c-478e-97d5-4f343aaa9ad4-plugins-config-volume\") pod \"14e890f7-843c-478e-97d5-4f343aaa9ad4\" (UID: \"14e890f7-843c-478e-97d5-4f343aaa9ad4\") "
Nov 02 13:29:36.257631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.257616    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ae8f04-3aba-4de2-9987-4f4bc895e558-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ae8f04-3aba-4de2-9987-4f4bc895e558-plugins-config-volume\") pod \"97ae8f04-3aba-4de2-9987-4f4bc895e558\" (UID: \"97ae8f04-3aba-4de2-9987-4f4bc895e558\") "
Nov 02 13:29:36.258013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.257967    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e890f7-843c-478e-97d5-4f343aaa9ad4-plugins-config-volume" pod "14e890f7-843c-478e-97d5-4f343aaa9ad4" (UID: "14e890f7-843c-478e-97d5-4f343aaa9ad4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:36.258387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.258362    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ae8f04-3aba-4de2-9987-4f4bc895e558-plugins-config-volume" pod "97ae8f04-3aba-4de2-9987-4f4bc895e558" (UID: "97ae8f04-3aba-4de2-9987-4f4bc895e558"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:36.258571 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.258551    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14e890f7-843c-478e-97d5-4f343aaa9ad4-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:36.258646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.258577    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97ae8f04-3aba-4de2-9987-4f4bc895e558-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:36.259799 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.259767    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ae8f04-3aba-4de2-9987-4f4bc895e558-kube-api-access-bj4bw" pod "97ae8f04-3aba-4de2-9987-4f4bc895e558" (UID: "97ae8f04-3aba-4de2-9987-4f4bc895e558"). InnerVolumeSpecName "kube-api-access-bj4bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:36.259944 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.259813    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e890f7-843c-478e-97d5-4f343aaa9ad4-kube-api-access-vqd8r" pod "14e890f7-843c-478e-97d5-4f343aaa9ad4" (UID: "14e890f7-843c-478e-97d5-4f343aaa9ad4"). InnerVolumeSpecName "kube-api-access-vqd8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:36.360484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.360449    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj4bw\" (UniqueName: \"kubernetes.io/projected/97ae8f04-3aba-4de2-9987-4f4bc895e558-kube-api-access-bj4bw\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:36.360484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.360478    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqd8r\" (UniqueName: \"kubernetes.io/projected/14e890f7-843c-478e-97d5-4f343aaa9ad4-kube-api-access-vqd8r\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:36.468354 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.468264    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7" podUID="aa872acf-f31b-45f6-bd50-629153ec892b" containerName="payload-processing" containerID="cri-o://5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8" gracePeriod=25
Nov 02 13:29:36.524435 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.524389    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj" podUID="bfb8420b-2f5d-4d05-81c5-850259014a61" containerName="payload-pre-processing" containerID="cri-o://2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b" gracePeriod=25
Nov 02 13:29:36.702509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.702473    5879 generic.go:356] "Generic (PLEG): container finished" podID="97ae8f04-3aba-4de2-9987-4f4bc895e558" containerID="72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1" exitCode=0
Nov 02 13:29:36.702732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.702517    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"
Nov 02 13:29:36.702732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.702560    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg" event={"ID":"97ae8f04-3aba-4de2-9987-4f4bc895e558","Type":"ContainerDied","Data":"72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1"}
Nov 02 13:29:36.702732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.702623    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg" event={"ID":"97ae8f04-3aba-4de2-9987-4f4bc895e558","Type":"ContainerDied","Data":"40ff928dbf4322f159a05134b98de0315027f4292b2635fc66a78b5372709433"}
Nov 02 13:29:36.702732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.702652    5879 scope.go:122] "RemoveContainer" containerID="72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1"
Nov 02 13:29:36.704282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.704258    5879 generic.go:356] "Generic (PLEG): container finished" podID="14e890f7-843c-478e-97d5-4f343aaa9ad4" containerID="44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a" exitCode=0
Nov 02 13:29:36.704406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.704296    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm" event={"ID":"14e890f7-843c-478e-97d5-4f343aaa9ad4","Type":"ContainerDied","Data":"44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a"}
Nov 02 13:29:36.704406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.704354    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm" event={"ID":"14e890f7-843c-478e-97d5-4f343aaa9ad4","Type":"ContainerDied","Data":"2381c4216d93d9987a87bef4967c5af4915da392b93b6586a7084767ae42acb5"}
Nov 02 13:29:36.704406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.704309    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"
Nov 02 13:29:36.716199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.716178    5879 scope.go:122] "RemoveContainer" containerID="72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1"
Nov 02 13:29:36.716471 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:29:36.716452    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1\": container with ID starting with 72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1 not found: ID does not exist" containerID="72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1"
Nov 02 13:29:36.716560 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.716480    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1"} err="failed to get container status \"72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1\": rpc error: code = NotFound desc = could not find container \"72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1\": container with ID starting with 72f26ed78aee9b6caff246f0cb5533cadf982507009129b5bdd7e9ca9d66efb1 not found: ID does not exist"
Nov 02 13:29:36.716560 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.716499    5879 scope.go:122] "RemoveContainer" containerID="44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a"
Nov 02 13:29:36.727273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.727249    5879 scope.go:122] "RemoveContainer" containerID="44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a"
Nov 02 13:29:36.727574 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:29:36.727554    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a\": container with ID starting with 44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a not found: ID does not exist" containerID="44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a"
Nov 02 13:29:36.727658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.727585    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a"} err="failed to get container status \"44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a\": rpc error: code = NotFound desc = could not find container \"44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a\": container with ID starting with 44fb1ea5d9db14368a5e176b4275823bbf1adc92cb90bd79b85eb9ab3659032a not found: ID does not exist"
Nov 02 13:29:36.760975 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.760944    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"]
Nov 02 13:29:36.774917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.774892    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-035dac-b889bf555-sgpxg"]
Nov 02 13:29:36.821741 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.821679    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"]
Nov 02 13:29:36.821919 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:36.821855    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-035dac-77f9c676f5-8c5qm"]
Nov 02 13:29:37.367463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:37.367429    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:37.567130 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:37.567094    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:38.008452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:38.008419    5879 scope.go:122] "RemoveContainer" containerID="cd49469781c59dea57f3e474f9579980bf33450b5b9d8678340152b7d6650d20"
Nov 02 13:29:38.020641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:38.020612    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e890f7-843c-478e-97d5-4f343aaa9ad4" path="/var/lib/kubelet/pods/14e890f7-843c-478e-97d5-4f343aaa9ad4/volumes"
Nov 02 13:29:38.021228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:38.021211    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ae8f04-3aba-4de2-9987-4f4bc895e558" path="/var/lib/kubelet/pods/97ae8f04-3aba-4de2-9987-4f4bc895e558/volumes"
Nov 02 13:29:39.282214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:39.282179    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:41.109029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:41.108985    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:41.277190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:41.277153    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:42.028321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:42.028285    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7" podUID="aa872acf-f31b-45f6-bd50-629153ec892b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:29:42.028527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:42.028341    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj" podUID="bfb8420b-2f5d-4d05-81c5-850259014a61" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:29:42.896889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:42.896847    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:44.677752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:44.677712    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:44.822076 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:29:44.822043    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:29:45.131223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:45.131195    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:45.274029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:45.273983    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:45.967040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:45.967010    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:46.405438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:46.405390    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:46.902527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:46.902490    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:47.107055 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.106982    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:47.447128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.447090    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"]
Nov 02 13:29:47.448031 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.448007    5879 cpu_manager.go:414] "containerMap: removing container" podUID="111b30d0-5386-4fc5-bc65-6108173dede5" containerName="payload-pre-processing"
Nov 02 13:29:47.448154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.448034    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111b30d0-5386-4fc5-bc65-6108173dede5" containerName="payload-pre-processing"
Nov 02 13:29:47.448154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.448052    5879 cpu_manager.go:414] "containerMap: removing container" podUID="2117da57-e909-43d4-a471-c0f4b3c0f13d" containerName="payload-processing"
Nov 02 13:29:47.448154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.448058    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2117da57-e909-43d4-a471-c0f4b3c0f13d" containerName="payload-processing"
Nov 02 13:29:47.448154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.448081    5879 cpu_manager.go:414] "containerMap: removing container" podUID="97ae8f04-3aba-4de2-9987-4f4bc895e558" containerName="payload-processing"
Nov 02 13:29:47.448154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.448086    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ae8f04-3aba-4de2-9987-4f4bc895e558" containerName="payload-processing"
Nov 02 13:29:47.448154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.448113    5879 cpu_manager.go:414] "containerMap: removing container" podUID="14e890f7-843c-478e-97d5-4f343aaa9ad4" containerName="payload-pre-processing"
Nov 02 13:29:47.448154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.448118    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e890f7-843c-478e-97d5-4f343aaa9ad4" containerName="payload-pre-processing"
Nov 02 13:29:47.448397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.448240    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="111b30d0-5386-4fc5-bc65-6108173dede5" containerName="payload-pre-processing"
Nov 02 13:29:47.448397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.448249    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="2117da57-e909-43d4-a471-c0f4b3c0f13d" containerName="payload-processing"
Nov 02 13:29:47.448397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.448258    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ae8f04-3aba-4de2-9987-4f4bc895e558" containerName="payload-processing"
Nov 02 13:29:47.448397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.448268    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e890f7-843c-478e-97d5-4f343aaa9ad4" containerName="payload-pre-processing"
Nov 02 13:29:47.453249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.453231    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:47.470669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.470648    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:29:47.484035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.484009    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:29:47.489533 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.489503    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"]
Nov 02 13:29:47.523701 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.523663    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-253c20b6-bspjw\" (UID: \"6f574761-e825-4e38-a1d6-3bd749376635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:47.523886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.523839    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-253c20b6-bspjw\" (UID: \"6f574761-e825-4e38-a1d6-3bd749376635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:47.523944 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.523923    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d7s2\" (UniqueName: \"kubernetes.io/projected/6f574761-e825-4e38-a1d6-3bd749376635-kube-api-access-2d7s2\") pod \"maas-api-revoke-sub-253c20b6-bspjw\" (UID: \"6f574761-e825-4e38-a1d6-3bd749376635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:47.625920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.625878    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-253c20b6-bspjw\" (UID: \"6f574761-e825-4e38-a1d6-3bd749376635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:47.626130 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.626028    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-253c20b6-bspjw\" (UID: \"6f574761-e825-4e38-a1d6-3bd749376635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:47.626130 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.626101    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d7s2\" (UniqueName: \"kubernetes.io/projected/6f574761-e825-4e38-a1d6-3bd749376635-kube-api-access-2d7s2\") pod \"maas-api-revoke-sub-253c20b6-bspjw\" (UID: \"6f574761-e825-4e38-a1d6-3bd749376635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:47.626867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.626839    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-253c20b6-bspjw\" (UID: \"6f574761-e825-4e38-a1d6-3bd749376635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:47.628590 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.628568    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-253c20b6-bspjw\" (UID: \"6f574761-e825-4e38-a1d6-3bd749376635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:47.652882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.652853    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d7s2\" (UniqueName: \"kubernetes.io/projected/6f574761-e825-4e38-a1d6-3bd749376635-kube-api-access-2d7s2\") pod \"maas-api-revoke-sub-253c20b6-bspjw\" (UID: \"6f574761-e825-4e38-a1d6-3bd749376635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:47.767669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:47.767566    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:48.083245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:48.083212    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"]
Nov 02 13:29:48.083691 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:29:48.083664    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f574761_e825_4e38_a1d6_3bd749376635.slice/crio-8ad5de828d2aa97864423376e197c297626542bf9dcac57e69b47b32391801ca WatchSource:0}: Error finding container 8ad5de828d2aa97864423376e197c297626542bf9dcac57e69b47b32391801ca: Status 404 returned error can't find the container with id 8ad5de828d2aa97864423376e197c297626542bf9dcac57e69b47b32391801ca
Nov 02 13:29:48.796683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:48.796592    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw" event={"ID":"6f574761-e825-4e38-a1d6-3bd749376635","Type":"ContainerStarted","Data":"98298fb6ee52ee5bd3d9edc978641dbad6719577fac29df34f5decd7d1a4c8bb"}
Nov 02 13:29:48.796683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:48.796662    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw" event={"ID":"6f574761-e825-4e38-a1d6-3bd749376635","Type":"ContainerStarted","Data":"8ad5de828d2aa97864423376e197c297626542bf9dcac57e69b47b32391801ca"}
Nov 02 13:29:48.840517 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:48.840454    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw" podStartSLOduration=1.8404396840000001 podStartE2EDuration="1.840439684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:29:48.839864093 +0000 UTC m=+3734.946566946" watchObservedRunningTime="2026-11-02 13:29:48.840439684 +0000 UTC m=+3734.947142534"
Nov 02 13:29:48.884323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:48.884280    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:49.805351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:49.805315    5879 generic.go:356] "Generic (PLEG): container finished" podID="6f574761-e825-4e38-a1d6-3bd749376635" containerID="98298fb6ee52ee5bd3d9edc978641dbad6719577fac29df34f5decd7d1a4c8bb" exitCode=0
Nov 02 13:29:49.805780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:49.805404    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw" event={"ID":"6f574761-e825-4e38-a1d6-3bd749376635","Type":"ContainerDied","Data":"98298fb6ee52ee5bd3d9edc978641dbad6719577fac29df34f5decd7d1a4c8bb"}
Nov 02 13:29:50.680044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:50.679979    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:50.984133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:50.984107    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:51.077822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.077785    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f574761-e825-4e38-a1d6-3bd749376635-kube-api-access-2d7s2\" (UniqueName: \"kubernetes.io/projected/6f574761-e825-4e38-a1d6-3bd749376635-kube-api-access-2d7s2\") pod \"6f574761-e825-4e38-a1d6-3bd749376635\" (UID: \"6f574761-e825-4e38-a1d6-3bd749376635\") "
Nov 02 13:29:51.078029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.077909    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-xks-ca\") pod \"6f574761-e825-4e38-a1d6-3bd749376635\" (UID: \"6f574761-e825-4e38-a1d6-3bd749376635\") "
Nov 02 13:29:51.078029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.077949    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-openshift-ca\") pod \"6f574761-e825-4e38-a1d6-3bd749376635\" (UID: \"6f574761-e825-4e38-a1d6-3bd749376635\") "
Nov 02 13:29:51.078337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.078312    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-openshift-ca" pod "6f574761-e825-4e38-a1d6-3bd749376635" (UID: "6f574761-e825-4e38-a1d6-3bd749376635"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:51.078497 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.078484    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:51.080304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.080286    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-xks-ca" pod "6f574761-e825-4e38-a1d6-3bd749376635" (UID: "6f574761-e825-4e38-a1d6-3bd749376635"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:29:51.080423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.080372    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f574761-e825-4e38-a1d6-3bd749376635-kube-api-access-2d7s2" pod "6f574761-e825-4e38-a1d6-3bd749376635" (UID: "6f574761-e825-4e38-a1d6-3bd749376635"). InnerVolumeSpecName "kube-api-access-2d7s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:51.182110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.181347    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f574761-e825-4e38-a1d6-3bd749376635-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:51.182110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.181385    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d7s2\" (UniqueName: \"kubernetes.io/projected/6f574761-e825-4e38-a1d6-3bd749376635-kube-api-access-2d7s2\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:51.821490 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.821450    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"
Nov 02 13:29:51.821688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.821485    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw" event={"ID":"6f574761-e825-4e38-a1d6-3bd749376635","Type":"ContainerDied","Data":"8ad5de828d2aa97864423376e197c297626542bf9dcac57e69b47b32391801ca"}
Nov 02 13:29:51.821688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.821579    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ad5de828d2aa97864423376e197c297626542bf9dcac57e69b47b32391801ca"
Nov 02 13:29:51.835579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.835550    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"]
Nov 02 13:29:51.849168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.849140    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"]
Nov 02 13:29:51.964067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:51.964037    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:52.028189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:52.028152    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj" podUID="bfb8420b-2f5d-4d05-81c5-850259014a61" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:29:52.028618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:52.028158    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7" podUID="aa872acf-f31b-45f6-bd50-629153ec892b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:29:52.544232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:52.544191    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:52.887442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:52.887401    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:54.068304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:54.068268    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"]
Nov 02 13:29:54.071291 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:54.071266    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-162d65ca-hnpzc"]
Nov 02 13:29:54.360345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:54.360259    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:56.020078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.020045    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="050c1f0a-55d3-4e78-a9f5-51b0652bc53c" path="/var/lib/kubelet/pods/050c1f0a-55d3-4e78-a9f5-51b0652bc53c/volumes"
Nov 02 13:29:56.139918 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.139844    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:56.818916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.818886    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"
Nov 02 13:29:56.836746 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.836719    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"
Nov 02 13:29:56.859767 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.859731    5879 generic.go:356] "Generic (PLEG): container finished" podID="bfb8420b-2f5d-4d05-81c5-850259014a61" containerID="2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b" exitCode=0
Nov 02 13:29:56.859981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.859785    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"
Nov 02 13:29:56.859981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.859809    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj" event={"ID":"bfb8420b-2f5d-4d05-81c5-850259014a61","Type":"ContainerDied","Data":"2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b"}
Nov 02 13:29:56.859981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.859880    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj" event={"ID":"bfb8420b-2f5d-4d05-81c5-850259014a61","Type":"ContainerDied","Data":"b5efafea981afb96086c4adb9c8a676fd5e7fc0188cec2f44edc8464710b1ef5"}
Nov 02 13:29:56.859981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.859908    5879 scope.go:122] "RemoveContainer" containerID="2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b"
Nov 02 13:29:56.861088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.860903    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb8420b-2f5d-4d05-81c5-850259014a61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb8420b-2f5d-4d05-81c5-850259014a61-plugins-config-volume\") pod \"bfb8420b-2f5d-4d05-81c5-850259014a61\" (UID: \"bfb8420b-2f5d-4d05-81c5-850259014a61\") "
Nov 02 13:29:56.861088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.860936    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa872acf-f31b-45f6-bd50-629153ec892b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa872acf-f31b-45f6-bd50-629153ec892b-plugins-config-volume\") pod \"aa872acf-f31b-45f6-bd50-629153ec892b\" (UID: \"aa872acf-f31b-45f6-bd50-629153ec892b\") "
Nov 02 13:29:56.861088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.861061    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa872acf-f31b-45f6-bd50-629153ec892b-kube-api-access-8fbdj\" (UniqueName: \"kubernetes.io/projected/aa872acf-f31b-45f6-bd50-629153ec892b-kube-api-access-8fbdj\") pod \"aa872acf-f31b-45f6-bd50-629153ec892b\" (UID: \"aa872acf-f31b-45f6-bd50-629153ec892b\") "
Nov 02 13:29:56.861331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.861191    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb8420b-2f5d-4d05-81c5-850259014a61-kube-api-access-vwrsd\" (UniqueName: \"kubernetes.io/projected/bfb8420b-2f5d-4d05-81c5-850259014a61-kube-api-access-vwrsd\") pod \"bfb8420b-2f5d-4d05-81c5-850259014a61\" (UID: \"bfb8420b-2f5d-4d05-81c5-850259014a61\") "
Nov 02 13:29:56.861444 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.861421    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb8420b-2f5d-4d05-81c5-850259014a61-plugins-config-volume" pod "bfb8420b-2f5d-4d05-81c5-850259014a61" (UID: "bfb8420b-2f5d-4d05-81c5-850259014a61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:56.861695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.861666    5879 generic.go:356] "Generic (PLEG): container finished" podID="aa872acf-f31b-45f6-bd50-629153ec892b" containerID="5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8" exitCode=0
Nov 02 13:29:56.861802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.861726    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"
Nov 02 13:29:56.861865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.861852    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7" event={"ID":"aa872acf-f31b-45f6-bd50-629153ec892b","Type":"ContainerDied","Data":"5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8"}
Nov 02 13:29:56.862124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.861894    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7" event={"ID":"aa872acf-f31b-45f6-bd50-629153ec892b","Type":"ContainerDied","Data":"04b22aec6a0aace7307bb24e372aad5ccfc7ca0b6c2963d8d1cf87561ea4cfdf"}
Nov 02 13:29:56.862124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.861951    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfb8420b-2f5d-4d05-81c5-850259014a61-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:56.862261 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.862167    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa872acf-f31b-45f6-bd50-629153ec892b-plugins-config-volume" pod "aa872acf-f31b-45f6-bd50-629153ec892b" (UID: "aa872acf-f31b-45f6-bd50-629153ec892b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:29:56.864344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.864312    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb8420b-2f5d-4d05-81c5-850259014a61-kube-api-access-vwrsd" pod "bfb8420b-2f5d-4d05-81c5-850259014a61" (UID: "bfb8420b-2f5d-4d05-81c5-850259014a61"). InnerVolumeSpecName "kube-api-access-vwrsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:56.864344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.864327    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa872acf-f31b-45f6-bd50-629153ec892b-kube-api-access-8fbdj" pod "aa872acf-f31b-45f6-bd50-629153ec892b" (UID: "aa872acf-f31b-45f6-bd50-629153ec892b"). InnerVolumeSpecName "kube-api-access-8fbdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:29:56.872913 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.872887    5879 scope.go:122] "RemoveContainer" containerID="2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b"
Nov 02 13:29:56.873369 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:29:56.873336    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b\": container with ID starting with 2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b not found: ID does not exist" containerID="2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b"
Nov 02 13:29:56.873486 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.873381    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b"} err="failed to get container status \"2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b\": rpc error: code = NotFound desc = could not find container \"2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b\": container with ID starting with 2a54dbafb66068285ff1737e3510f9ed5c765f2389157a13a094a9f2e69efb3b not found: ID does not exist"
Nov 02 13:29:56.873486 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.873408    5879 scope.go:122] "RemoveContainer" containerID="5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8"
Nov 02 13:29:56.885623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.885599    5879 scope.go:122] "RemoveContainer" containerID="5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8"
Nov 02 13:29:56.886029 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:29:56.885967    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8\": container with ID starting with 5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8 not found: ID does not exist" containerID="5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8"
Nov 02 13:29:56.886123 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.886041    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8"} err="failed to get container status \"5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8\": rpc error: code = NotFound desc = could not find container \"5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8\": container with ID starting with 5328d8ef99b12ef95a1a2b3ab32680d2227696bc97b98ddd435775e1a5a646a8 not found: ID does not exist"
Nov 02 13:29:56.963461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.963424    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwrsd\" (UniqueName: \"kubernetes.io/projected/bfb8420b-2f5d-4d05-81c5-850259014a61-kube-api-access-vwrsd\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:56.963461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.963453    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa872acf-f31b-45f6-bd50-629153ec892b-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:56.963461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:56.963463    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fbdj\" (UniqueName: \"kubernetes.io/projected/aa872acf-f31b-45f6-bd50-629153ec892b-kube-api-access-8fbdj\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:29:57.213683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:57.213646    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"]
Nov 02 13:29:57.216901 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:57.216873    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-00d004-768498cbc9-xv4lj"]
Nov 02 13:29:57.276136 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:57.276101    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"]
Nov 02 13:29:57.281091 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:57.281064    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-00d004-5dd8cc5b85-sgdv7"]
Nov 02 13:29:57.965471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:57.965441    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:58.017694 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.017660    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa872acf-f31b-45f6-bd50-629153ec892b" path="/var/lib/kubelet/pods/aa872acf-f31b-45f6-bd50-629153ec892b/volumes"
Nov 02 13:29:58.018109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.018095    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb8420b-2f5d-4d05-81c5-850259014a61" path="/var/lib/kubelet/pods/bfb8420b-2f5d-4d05-81c5-850259014a61/volumes"
Nov 02 13:29:58.806346 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.806311    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"]
Nov 02 13:29:58.807393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.807370    5879 cpu_manager.go:414] "containerMap: removing container" podUID="6f574761-e825-4e38-a1d6-3bd749376635" containerName="revoke-keys"
Nov 02 13:29:58.807443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.807396    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f574761-e825-4e38-a1d6-3bd749376635" containerName="revoke-keys"
Nov 02 13:29:58.807443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.807414    5879 cpu_manager.go:414] "containerMap: removing container" podUID="bfb8420b-2f5d-4d05-81c5-850259014a61" containerName="payload-pre-processing"
Nov 02 13:29:58.807443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.807423    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb8420b-2f5d-4d05-81c5-850259014a61" containerName="payload-pre-processing"
Nov 02 13:29:58.807443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.807439    5879 cpu_manager.go:414] "containerMap: removing container" podUID="aa872acf-f31b-45f6-bd50-629153ec892b" containerName="payload-processing"
Nov 02 13:29:58.807605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.807448    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa872acf-f31b-45f6-bd50-629153ec892b" containerName="payload-processing"
Nov 02 13:29:58.807642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.807633    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f574761-e825-4e38-a1d6-3bd749376635" containerName="revoke-keys"
Nov 02 13:29:58.807677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.807645    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb8420b-2f5d-4d05-81c5-850259014a61" containerName="payload-pre-processing"
Nov 02 13:29:58.807677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.807654    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa872acf-f31b-45f6-bd50-629153ec892b" containerName="payload-processing"
Nov 02 13:29:58.813125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.813102    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:29:58.830943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.830914    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"]
Nov 02 13:29:58.840216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.840186    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:29:58.840416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.840363    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:29:58.888585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.888550    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-385ad4f2-zz8gq\" (UID: \"048eb183-2cd4-49fc-b6fc-2c4662a29637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:29:58.888585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.888591    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhmzg\" (UniqueName: \"kubernetes.io/projected/048eb183-2cd4-49fc-b6fc-2c4662a29637-kube-api-access-mhmzg\") pod \"maas-api-revoke-sub-385ad4f2-zz8gq\" (UID: \"048eb183-2cd4-49fc-b6fc-2c4662a29637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:29:58.888819 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.888676    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-385ad4f2-zz8gq\" (UID: \"048eb183-2cd4-49fc-b6fc-2c4662a29637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:29:58.990606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.990576    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-385ad4f2-zz8gq\" (UID: \"048eb183-2cd4-49fc-b6fc-2c4662a29637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:29:58.990779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.990616    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhmzg\" (UniqueName: \"kubernetes.io/projected/048eb183-2cd4-49fc-b6fc-2c4662a29637-kube-api-access-mhmzg\") pod \"maas-api-revoke-sub-385ad4f2-zz8gq\" (UID: \"048eb183-2cd4-49fc-b6fc-2c4662a29637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:29:58.990779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.990670    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-385ad4f2-zz8gq\" (UID: \"048eb183-2cd4-49fc-b6fc-2c4662a29637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:29:58.991331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.991301    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-385ad4f2-zz8gq\" (UID: \"048eb183-2cd4-49fc-b6fc-2c4662a29637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:29:58.993265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:58.993238    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-385ad4f2-zz8gq\" (UID: \"048eb183-2cd4-49fc-b6fc-2c4662a29637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:29:59.016268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:59.016239    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhmzg\" (UniqueName: \"kubernetes.io/projected/048eb183-2cd4-49fc-b6fc-2c4662a29637-kube-api-access-mhmzg\") pod \"maas-api-revoke-sub-385ad4f2-zz8gq\" (UID: \"048eb183-2cd4-49fc-b6fc-2c4662a29637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:29:59.128548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:59.128518    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:29:59.440117 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:29:59.440087    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048eb183_2cd4_49fc_b6fc_2c4662a29637.slice/crio-c1fdf10420715967a25d1a7f4dfaca13ebc631fa3502fd6a38c9c2ab02409c91 WatchSource:0}: Error finding container c1fdf10420715967a25d1a7f4dfaca13ebc631fa3502fd6a38c9c2ab02409c91: Status 404 returned error can't find the container with id c1fdf10420715967a25d1a7f4dfaca13ebc631fa3502fd6a38c9c2ab02409c91
Nov 02 13:29:59.441199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:59.441174    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"]
Nov 02 13:29:59.793371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:59.793338    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:29:59.900257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:59.900193    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq" event={"ID":"048eb183-2cd4-49fc-b6fc-2c4662a29637","Type":"ContainerStarted","Data":"b384542edea209b2fb0319c990e5b54f206fb91f8ccf9f46a439a2b3e92b0e4f"}
Nov 02 13:29:59.900722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:59.900347    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq" event={"ID":"048eb183-2cd4-49fc-b6fc-2c4662a29637","Type":"ContainerStarted","Data":"c1fdf10420715967a25d1a7f4dfaca13ebc631fa3502fd6a38c9c2ab02409c91"}
Nov 02 13:29:59.950240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:29:59.950125    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq" podStartSLOduration=1.950108566 podStartE2EDuration="1.950108566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:29:59.948083091 +0000 UTC m=+3746.054785942" watchObservedRunningTime="2026-11-02 13:29:59.950108566 +0000 UTC m=+3746.056811414"
Nov 02 13:30:00.084604 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.084571    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:00.173612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.169800    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"]
Nov 02 13:30:00.178304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.178267    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"
Nov 02 13:30:00.179112 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.179083    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"]
Nov 02 13:30:00.185962 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.185935    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"]
Nov 02 13:30:00.186640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.186612    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"
Nov 02 13:30:00.191225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.190943    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"]
Nov 02 13:30:00.191225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.191096    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"
Nov 02 13:30:00.197903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.197877    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"]
Nov 02 13:30:00.198250 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.197985    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"
Nov 02 13:30:00.203871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.203732    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"
Nov 02 13:30:00.205390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.205359    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"]
Nov 02 13:30:00.208300 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.208271    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8xbk\" (UniqueName: \"kubernetes.io/projected/299f06eb-e0df-4927-9d60-738e88a03192-kube-api-access-v8xbk\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw\" (UID: \"299f06eb-e0df-4927-9d60-738e88a03192\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"
Nov 02 13:30:00.208432 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.208409    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/299f06eb-e0df-4927-9d60-738e88a03192-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw\" (UID: \"299f06eb-e0df-4927-9d60-738e88a03192\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"
Nov 02 13:30:00.210130 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.210106    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"]
Nov 02 13:30:00.220074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.220037    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"]
Nov 02 13:30:00.220271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.220110    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"]
Nov 02 13:30:00.225477 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.225441    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"]
Nov 02 13:30:00.230103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.230080    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"
Nov 02 13:30:00.230794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.230734    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"]
Nov 02 13:30:00.236097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.236040    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"]
Nov 02 13:30:00.236260 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.236104    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"
Nov 02 13:30:00.242587 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.242565    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"]
Nov 02 13:30:00.243548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.243524    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"
Nov 02 13:30:00.248610 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.248582    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"
Nov 02 13:30:00.253437 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.253408    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"]
Nov 02 13:30:00.272940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.272914    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"]
Nov 02 13:30:00.290051 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.286079    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"]
Nov 02 13:30:00.290051 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.289475    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:00.291524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.291499    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"]
Nov 02 13:30:00.296331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.296293    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"]
Nov 02 13:30:00.307520 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.307193    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"]
Nov 02 13:30:00.307520 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.307316    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"
Nov 02 13:30:00.317110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.311508    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226sd\" (UniqueName: \"kubernetes.io/projected/09b683bc-953d-46d1-bc3b-104adddc15f7-kube-api-access-226sd\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh\" (UID: \"09b683bc-953d-46d1-bc3b-104adddc15f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"
Nov 02 13:30:00.317110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.311608    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8xbk\" (UniqueName: \"kubernetes.io/projected/299f06eb-e0df-4927-9d60-738e88a03192-kube-api-access-v8xbk\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw\" (UID: \"299f06eb-e0df-4927-9d60-738e88a03192\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"
Nov 02 13:30:00.317110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.311956    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzcrb\" (UniqueName: \"kubernetes.io/projected/5e755e54-3856-4cab-818e-36e2aca4076d-kube-api-access-pzcrb\") pod \"maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt\" (UID: \"5e755e54-3856-4cab-818e-36e2aca4076d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"
Nov 02 13:30:00.317110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.312825    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd1e90e-1165-4bde-8d01-3d37aee5802e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5\" (UID: \"acd1e90e-1165-4bde-8d01-3d37aee5802e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"
Nov 02 13:30:00.317110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.316335    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"]
Nov 02 13:30:00.317110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317069    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7bgj\" (UniqueName: \"kubernetes.io/projected/acd1e90e-1165-4bde-8d01-3d37aee5802e-kube-api-access-r7bgj\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5\" (UID: \"acd1e90e-1165-4bde-8d01-3d37aee5802e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317123    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90197636-9fa4-4217-9aaf-7148e5b1479f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p\" (UID: \"90197636-9fa4-4217-9aaf-7148e5b1479f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317168    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/299f06eb-e0df-4927-9d60-738e88a03192-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw\" (UID: \"299f06eb-e0df-4927-9d60-738e88a03192\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317265    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe86781-bd96-4318-8447-767533806fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr\" (UID: \"ebe86781-bd96-4318-8447-767533806fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317339    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8\" (UID: \"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317391    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357161b9-27d9-4b14-81de-365061ce42f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj\" (UID: \"357161b9-27d9-4b14-81de-365061ce42f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317440    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e755e54-3856-4cab-818e-36e2aca4076d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt\" (UID: \"5e755e54-3856-4cab-818e-36e2aca4076d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317465    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7g5h\" (UniqueName: \"kubernetes.io/projected/ebe86781-bd96-4318-8447-767533806fce-kube-api-access-q7g5h\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr\" (UID: \"ebe86781-bd96-4318-8447-767533806fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317529    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nntst\" (UniqueName: \"kubernetes.io/projected/357161b9-27d9-4b14-81de-365061ce42f6-kube-api-access-nntst\") pod \"maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj\" (UID: \"357161b9-27d9-4b14-81de-365061ce42f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317592    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d463a6e-b5f3-407e-8b41-6836c3b709f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb\" (UID: \"3d463a6e-b5f3-407e-8b41-6836c3b709f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317621    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09b683bc-953d-46d1-bc3b-104adddc15f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh\" (UID: \"09b683bc-953d-46d1-bc3b-104adddc15f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317755    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw925\" (UniqueName: \"kubernetes.io/projected/90197636-9fa4-4217-9aaf-7148e5b1479f-kube-api-access-rw925\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p\" (UID: \"90197636-9fa4-4217-9aaf-7148e5b1479f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317788    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dstg6\" (UniqueName: \"kubernetes.io/projected/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-kube-api-access-dstg6\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8\" (UID: \"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.317858    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd4vz\" (UniqueName: \"kubernetes.io/projected/3d463a6e-b5f3-407e-8b41-6836c3b709f1-kube-api-access-kd4vz\") pod \"maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb\" (UID: \"3d463a6e-b5f3-407e-8b41-6836c3b709f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"
Nov 02 13:30:00.324507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.319450    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/299f06eb-e0df-4927-9d60-738e88a03192-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw\" (UID: \"299f06eb-e0df-4927-9d60-738e88a03192\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"
Nov 02 13:30:00.339391 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.329538    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"]
Nov 02 13:30:00.339391 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.330544    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"
Nov 02 13:30:00.348041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.345281    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"]
Nov 02 13:30:00.348041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.345343    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"]
Nov 02 13:30:00.348041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.345446    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"
Nov 02 13:30:00.354085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.354050    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"]
Nov 02 13:30:00.358978 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.358938    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8xbk\" (UniqueName: \"kubernetes.io/projected/299f06eb-e0df-4927-9d60-738e88a03192-kube-api-access-v8xbk\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw\" (UID: \"299f06eb-e0df-4927-9d60-738e88a03192\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"
Nov 02 13:30:00.419605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.419574    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8\" (UID: \"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"
Nov 02 13:30:00.419777 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.419624    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357161b9-27d9-4b14-81de-365061ce42f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj\" (UID: \"357161b9-27d9-4b14-81de-365061ce42f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"
Nov 02 13:30:00.419777 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.419675    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e755e54-3856-4cab-818e-36e2aca4076d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt\" (UID: \"5e755e54-3856-4cab-818e-36e2aca4076d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"
Nov 02 13:30:00.419777 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.419702    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7g5h\" (UniqueName: \"kubernetes.io/projected/ebe86781-bd96-4318-8447-767533806fce-kube-api-access-q7g5h\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr\" (UID: \"ebe86781-bd96-4318-8447-767533806fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"
Nov 02 13:30:00.419777 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.419760    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nntst\" (UniqueName: \"kubernetes.io/projected/357161b9-27d9-4b14-81de-365061ce42f6-kube-api-access-nntst\") pod \"maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj\" (UID: \"357161b9-27d9-4b14-81de-365061ce42f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"
Nov 02 13:30:00.420040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.419818    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d463a6e-b5f3-407e-8b41-6836c3b709f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb\" (UID: \"3d463a6e-b5f3-407e-8b41-6836c3b709f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"
Nov 02 13:30:00.420040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.419846    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09b683bc-953d-46d1-bc3b-104adddc15f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh\" (UID: \"09b683bc-953d-46d1-bc3b-104adddc15f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"
Nov 02 13:30:00.420040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.419882    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rnh4\" (UniqueName: \"kubernetes.io/projected/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-kube-api-access-2rnh4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7\" (UID: \"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"
Nov 02 13:30:00.420040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.419949    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a07f2c-977a-4740-b9fb-56ee22098f68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n\" (UID: \"74a07f2c-977a-4740-b9fb-56ee22098f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"
Nov 02 13:30:00.420040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420029    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw925\" (UniqueName: \"kubernetes.io/projected/90197636-9fa4-4217-9aaf-7148e5b1479f-kube-api-access-rw925\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p\" (UID: \"90197636-9fa4-4217-9aaf-7148e5b1479f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"
Nov 02 13:30:00.420344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420052    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstg6\" (UniqueName: \"kubernetes.io/projected/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-kube-api-access-dstg6\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8\" (UID: \"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"
Nov 02 13:30:00.420344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420101    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8h2\" (UniqueName: \"kubernetes.io/projected/74a07f2c-977a-4740-b9fb-56ee22098f68-kube-api-access-5k8h2\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n\" (UID: \"74a07f2c-977a-4740-b9fb-56ee22098f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"
Nov 02 13:30:00.420344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420139    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd4vz\" (UniqueName: \"kubernetes.io/projected/3d463a6e-b5f3-407e-8b41-6836c3b709f1-kube-api-access-kd4vz\") pod \"maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb\" (UID: \"3d463a6e-b5f3-407e-8b41-6836c3b709f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"
Nov 02 13:30:00.420344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420174    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226sd\" (UniqueName: \"kubernetes.io/projected/09b683bc-953d-46d1-bc3b-104adddc15f7-kube-api-access-226sd\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh\" (UID: \"09b683bc-953d-46d1-bc3b-104adddc15f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"
Nov 02 13:30:00.420344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420219    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzcrb\" (UniqueName: \"kubernetes.io/projected/5e755e54-3856-4cab-818e-36e2aca4076d-kube-api-access-pzcrb\") pod \"maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt\" (UID: \"5e755e54-3856-4cab-818e-36e2aca4076d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"
Nov 02 13:30:00.420344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420247    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd1e90e-1165-4bde-8d01-3d37aee5802e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5\" (UID: \"acd1e90e-1165-4bde-8d01-3d37aee5802e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"
Nov 02 13:30:00.420344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420279    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qtdf\" (UniqueName: \"kubernetes.io/projected/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-kube-api-access-9qtdf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k\" (UID: \"4a30bbaa-5aa8-469e-8c1d-1e58a821b310\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"
Nov 02 13:30:00.420344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420309    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7bgj\" (UniqueName: \"kubernetes.io/projected/acd1e90e-1165-4bde-8d01-3d37aee5802e-kube-api-access-r7bgj\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5\" (UID: \"acd1e90e-1165-4bde-8d01-3d37aee5802e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"
Nov 02 13:30:00.420802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420518    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8\" (UID: \"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"
Nov 02 13:30:00.420802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420706    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d463a6e-b5f3-407e-8b41-6836c3b709f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb\" (UID: \"3d463a6e-b5f3-407e-8b41-6836c3b709f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"
Nov 02 13:30:00.420802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420752    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e755e54-3856-4cab-818e-36e2aca4076d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt\" (UID: \"5e755e54-3856-4cab-818e-36e2aca4076d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"
Nov 02 13:30:00.420971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420828    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90197636-9fa4-4217-9aaf-7148e5b1479f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p\" (UID: \"90197636-9fa4-4217-9aaf-7148e5b1479f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"
Nov 02 13:30:00.420971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420899    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k\" (UID: \"4a30bbaa-5aa8-469e-8c1d-1e58a821b310\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"
Nov 02 13:30:00.420971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.420959    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7\" (UID: \"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"
Nov 02 13:30:00.421212 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.421186    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe86781-bd96-4318-8447-767533806fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr\" (UID: \"ebe86781-bd96-4318-8447-767533806fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"
Nov 02 13:30:00.421408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.421384    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09b683bc-953d-46d1-bc3b-104adddc15f7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh\" (UID: \"09b683bc-953d-46d1-bc3b-104adddc15f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"
Nov 02 13:30:00.421562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.421540    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90197636-9fa4-4217-9aaf-7148e5b1479f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p\" (UID: \"90197636-9fa4-4217-9aaf-7148e5b1479f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"
Nov 02 13:30:00.421647 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.421592    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd1e90e-1165-4bde-8d01-3d37aee5802e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5\" (UID: \"acd1e90e-1165-4bde-8d01-3d37aee5802e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"
Nov 02 13:30:00.421822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.421801    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe86781-bd96-4318-8447-767533806fce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr\" (UID: \"ebe86781-bd96-4318-8447-767533806fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"
Nov 02 13:30:00.421888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.421871    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357161b9-27d9-4b14-81de-365061ce42f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj\" (UID: \"357161b9-27d9-4b14-81de-365061ce42f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"
Nov 02 13:30:00.448314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.448283    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7g5h\" (UniqueName: \"kubernetes.io/projected/ebe86781-bd96-4318-8447-767533806fce-kube-api-access-q7g5h\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr\" (UID: \"ebe86781-bd96-4318-8447-767533806fce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"
Nov 02 13:30:00.455793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.455704    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nntst\" (UniqueName: \"kubernetes.io/projected/357161b9-27d9-4b14-81de-365061ce42f6-kube-api-access-nntst\") pod \"maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj\" (UID: \"357161b9-27d9-4b14-81de-365061ce42f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"
Nov 02 13:30:00.455961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.455869    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dstg6\" (UniqueName: \"kubernetes.io/projected/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-kube-api-access-dstg6\") pod \"maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8\" (UID: \"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"
Nov 02 13:30:00.456321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.456304    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd4vz\" (UniqueName: \"kubernetes.io/projected/3d463a6e-b5f3-407e-8b41-6836c3b709f1-kube-api-access-kd4vz\") pod \"maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb\" (UID: \"3d463a6e-b5f3-407e-8b41-6836c3b709f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"
Nov 02 13:30:00.459193 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.459169    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzcrb\" (UniqueName: \"kubernetes.io/projected/5e755e54-3856-4cab-818e-36e2aca4076d-kube-api-access-pzcrb\") pod \"maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt\" (UID: \"5e755e54-3856-4cab-818e-36e2aca4076d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"
Nov 02 13:30:00.461034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.461010    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw925\" (UniqueName: \"kubernetes.io/projected/90197636-9fa4-4217-9aaf-7148e5b1479f-kube-api-access-rw925\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p\" (UID: \"90197636-9fa4-4217-9aaf-7148e5b1479f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"
Nov 02 13:30:00.462375 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.462351    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-226sd\" (UniqueName: \"kubernetes.io/projected/09b683bc-953d-46d1-bc3b-104adddc15f7-kube-api-access-226sd\") pod \"maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh\" (UID: \"09b683bc-953d-46d1-bc3b-104adddc15f7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"
Nov 02 13:30:00.470370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.470348    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7bgj\" (UniqueName: \"kubernetes.io/projected/acd1e90e-1165-4bde-8d01-3d37aee5802e-kube-api-access-r7bgj\") pod \"maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5\" (UID: \"acd1e90e-1165-4bde-8d01-3d37aee5802e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"
Nov 02 13:30:00.507054 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.507016    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"
Nov 02 13:30:00.522222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.522187    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"
Nov 02 13:30:00.522723 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.522693    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8h2\" (UniqueName: \"kubernetes.io/projected/74a07f2c-977a-4740-b9fb-56ee22098f68-kube-api-access-5k8h2\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n\" (UID: \"74a07f2c-977a-4740-b9fb-56ee22098f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"
Nov 02 13:30:00.522836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.522781    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qtdf\" (UniqueName: \"kubernetes.io/projected/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-kube-api-access-9qtdf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k\" (UID: \"4a30bbaa-5aa8-469e-8c1d-1e58a821b310\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"
Nov 02 13:30:00.522836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.522823    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k\" (UID: \"4a30bbaa-5aa8-469e-8c1d-1e58a821b310\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"
Nov 02 13:30:00.522932 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.522872    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7\" (UID: \"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"
Nov 02 13:30:00.523114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.523099    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rnh4\" (UniqueName: \"kubernetes.io/projected/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-kube-api-access-2rnh4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7\" (UID: \"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"
Nov 02 13:30:00.523204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.523146    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a07f2c-977a-4740-b9fb-56ee22098f68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n\" (UID: \"74a07f2c-977a-4740-b9fb-56ee22098f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"
Nov 02 13:30:00.523797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.523669    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7\" (UID: \"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"
Nov 02 13:30:00.523797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.523786    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a07f2c-977a-4740-b9fb-56ee22098f68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n\" (UID: \"74a07f2c-977a-4740-b9fb-56ee22098f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"
Nov 02 13:30:00.523911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.523801    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k\" (UID: \"4a30bbaa-5aa8-469e-8c1d-1e58a821b310\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"
Nov 02 13:30:00.531652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.531626    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"
Nov 02 13:30:00.549987 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.549960    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rnh4\" (UniqueName: \"kubernetes.io/projected/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-kube-api-access-2rnh4\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7\" (UID: \"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"
Nov 02 13:30:00.550812 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.550787    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8h2\" (UniqueName: \"kubernetes.io/projected/74a07f2c-977a-4740-b9fb-56ee22098f68-kube-api-access-5k8h2\") pod \"maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n\" (UID: \"74a07f2c-977a-4740-b9fb-56ee22098f68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"
Nov 02 13:30:00.551260 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.551240    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qtdf\" (UniqueName: \"kubernetes.io/projected/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-kube-api-access-9qtdf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k\" (UID: \"4a30bbaa-5aa8-469e-8c1d-1e58a821b310\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"
Nov 02 13:30:00.552891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.552869    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"
Nov 02 13:30:00.567966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.567939    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"
Nov 02 13:30:00.581033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.575554    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"
Nov 02 13:30:00.597192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.595221    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"
Nov 02 13:30:00.613574 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.608095    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"
Nov 02 13:30:00.639684 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.637455    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"
Nov 02 13:30:00.713495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.713167    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"
Nov 02 13:30:00.726327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.722036    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"
Nov 02 13:30:00.738665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.730679    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"
Nov 02 13:30:00.929370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.929330    5879 generic.go:356] "Generic (PLEG): container finished" podID="048eb183-2cd4-49fc-b6fc-2c4662a29637" containerID="b384542edea209b2fb0319c990e5b54f206fb91f8ccf9f46a439a2b3e92b0e4f" exitCode=0
Nov 02 13:30:00.929370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:00.929371    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq" event={"ID":"048eb183-2cd4-49fc-b6fc-2c4662a29637","Type":"ContainerDied","Data":"b384542edea209b2fb0319c990e5b54f206fb91f8ccf9f46a439a2b3e92b0e4f"}
Nov 02 13:30:01.043676 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.043619    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"]
Nov 02 13:30:01.045749 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:30:01.045609    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299f06eb_e0df_4927_9d60_738e88a03192.slice/crio-5b8dd1ab8ac6b152edd8beb68ddb19d5592234317bc6977000c5022654276cfa WatchSource:0}: Error finding container 5b8dd1ab8ac6b152edd8beb68ddb19d5592234317bc6977000c5022654276cfa: Status 404 returned error can't find the container with id 5b8dd1ab8ac6b152edd8beb68ddb19d5592234317bc6977000c5022654276cfa
Nov 02 13:30:01.092393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.092356    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"]
Nov 02 13:30:01.096888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.096859    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"]
Nov 02 13:30:01.157652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.157619    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:01.193809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.193772    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"]
Nov 02 13:30:01.202012 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.201959    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"]
Nov 02 13:30:01.206078 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:30:01.205150    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d463a6e_b5f3_407e_8b41_6836c3b709f1.slice/crio-f5eb00961c24285d89d42fb3f0f82698a111cd98ddbe621b4397265cd71b017f WatchSource:0}: Error finding container f5eb00961c24285d89d42fb3f0f82698a111cd98ddbe621b4397265cd71b017f: Status 404 returned error can't find the container with id f5eb00961c24285d89d42fb3f0f82698a111cd98ddbe621b4397265cd71b017f
Nov 02 13:30:01.226499 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:30:01.226454    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e755e54_3856_4cab_818e_36e2aca4076d.slice/crio-db27f744a57a916671b64f50a9555f7421ad1a34ad3ff41cc58b81be0f4d6416 WatchSource:0}: Error finding container db27f744a57a916671b64f50a9555f7421ad1a34ad3ff41cc58b81be0f4d6416: Status 404 returned error can't find the container with id db27f744a57a916671b64f50a9555f7421ad1a34ad3ff41cc58b81be0f4d6416
Nov 02 13:30:01.286620 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.286519    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:01.330981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.330951    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"]
Nov 02 13:30:01.334282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.334259    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"]
Nov 02 13:30:01.337129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.337103    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"]
Nov 02 13:30:01.387537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.387502    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"]
Nov 02 13:30:01.401920 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:30:01.398786    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a07f2c_977a_4740_b9fb_56ee22098f68.slice/crio-1038d06905b36cb172a337a6159a4083af09d8900e99f2918297e6e2223078ef WatchSource:0}: Error finding container 1038d06905b36cb172a337a6159a4083af09d8900e99f2918297e6e2223078ef: Status 404 returned error can't find the container with id 1038d06905b36cb172a337a6159a4083af09d8900e99f2918297e6e2223078ef
Nov 02 13:30:01.401920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.401405    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"]
Nov 02 13:30:01.404077 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:30:01.404047    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b683bc_953d_46d1_bc3b_104adddc15f7.slice/crio-ccdc6096e0de556f0deb59af45066431271886b1955b77b2446208ac718fda83 WatchSource:0}: Error finding container ccdc6096e0de556f0deb59af45066431271886b1955b77b2446208ac718fda83: Status 404 returned error can't find the container with id ccdc6096e0de556f0deb59af45066431271886b1955b77b2446208ac718fda83
Nov 02 13:30:01.406292 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.405971    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"]
Nov 02 13:30:01.449742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.449714    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"]
Nov 02 13:30:01.939739 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.939705    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr" event={"ID":"ebe86781-bd96-4318-8447-767533806fce","Type":"ContainerStarted","Data":"f8bdabf86b8d60c429da72dbca7561cb0f3d96508cf0bdda8139d3309a9a5007"}
Nov 02 13:30:01.941230 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.941199    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8" event={"ID":"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136","Type":"ContainerStarted","Data":"459acfb42265480f8f9f020709c49e42e711e52bfd133c2aab3b4816132c848f"}
Nov 02 13:30:01.942809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.942783    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj" event={"ID":"357161b9-27d9-4b14-81de-365061ce42f6","Type":"ContainerStarted","Data":"de7638bf0747cf9f7b56e7fbac0c9e4bfe3651f6f0a1a69a99c8854e32318681"}
Nov 02 13:30:01.942924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.942830    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj" event={"ID":"357161b9-27d9-4b14-81de-365061ce42f6","Type":"ContainerStarted","Data":"f6e656daa654ef3a5f133a3752ee3d4e43f62bde9f0dfe24a7f64f91663f08cd"}
Nov 02 13:30:01.944270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.944236    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh" event={"ID":"09b683bc-953d-46d1-bc3b-104adddc15f7","Type":"ContainerStarted","Data":"ccdc6096e0de556f0deb59af45066431271886b1955b77b2446208ac718fda83"}
Nov 02 13:30:01.945465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.945440    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw" event={"ID":"299f06eb-e0df-4927-9d60-738e88a03192","Type":"ContainerStarted","Data":"5b8dd1ab8ac6b152edd8beb68ddb19d5592234317bc6977000c5022654276cfa"}
Nov 02 13:30:01.946577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.946556    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb" event={"ID":"3d463a6e-b5f3-407e-8b41-6836c3b709f1","Type":"ContainerStarted","Data":"f5eb00961c24285d89d42fb3f0f82698a111cd98ddbe621b4397265cd71b017f"}
Nov 02 13:30:01.947686 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.947666    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p" event={"ID":"90197636-9fa4-4217-9aaf-7148e5b1479f","Type":"ContainerStarted","Data":"8b91937230cb4b5e4736e1c2379c12c1bde6b649794ad58e95e2f6f21d437331"}
Nov 02 13:30:01.948867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.948837    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k" event={"ID":"4a30bbaa-5aa8-469e-8c1d-1e58a821b310","Type":"ContainerStarted","Data":"6fe245358e6e1a17fbc7c08f22bfce672a21927af476092c536aa07ed7c114ee"}
Nov 02 13:30:01.950505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.950480    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt" event={"ID":"5e755e54-3856-4cab-818e-36e2aca4076d","Type":"ContainerStarted","Data":"5b1375d1cc1c080784365ef8dddae6f89ecbd6ba095b24b1d69ba153ca0772c9"}
Nov 02 13:30:01.950584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.950525    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt" event={"ID":"5e755e54-3856-4cab-818e-36e2aca4076d","Type":"ContainerStarted","Data":"db27f744a57a916671b64f50a9555f7421ad1a34ad3ff41cc58b81be0f4d6416"}
Nov 02 13:30:01.951713 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.951693    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n" event={"ID":"74a07f2c-977a-4740-b9fb-56ee22098f68","Type":"ContainerStarted","Data":"1038d06905b36cb172a337a6159a4083af09d8900e99f2918297e6e2223078ef"}
Nov 02 13:30:01.953145 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.953126    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7" event={"ID":"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0","Type":"ContainerStarted","Data":"ab923398d0d1fb2da081d2bddfa276fb1a8ef414a139dfb239bc95831da35c62"}
Nov 02 13:30:01.954540 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.954517    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5" event={"ID":"acd1e90e-1165-4bde-8d01-3d37aee5802e","Type":"ContainerStarted","Data":"94d272c80931bce94d174e67cc499a77b0b9978ba8fdfa0c4ecb4636be27d5e8"}
Nov 02 13:30:01.984259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.984208    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:01.994174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:01.986049    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj" podStartSLOduration=1.986029396 podStartE2EDuration="1.986029396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:01.985182192 +0000 UTC m=+3748.091885046" watchObservedRunningTime="2026-11-02 13:30:01.986029396 +0000 UTC m=+3748.092732250"
Nov 02 13:30:02.030183 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.030105    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt" podStartSLOduration=2.030084243 podStartE2EDuration="2.030084243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:02.028874739 +0000 UTC m=+3748.135577590" watchObservedRunningTime="2026-11-02 13:30:02.030084243 +0000 UTC m=+3748.136787171"
Nov 02 13:30:02.338097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.338071    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:30:02.468898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.468813    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-openshift-ca\") pod \"048eb183-2cd4-49fc-b6fc-2c4662a29637\" (UID: \"048eb183-2cd4-49fc-b6fc-2c4662a29637\") "
Nov 02 13:30:02.469075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.468983    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-xks-ca\") pod \"048eb183-2cd4-49fc-b6fc-2c4662a29637\" (UID: \"048eb183-2cd4-49fc-b6fc-2c4662a29637\") "
Nov 02 13:30:02.469140 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.469110    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048eb183-2cd4-49fc-b6fc-2c4662a29637-kube-api-access-mhmzg\" (UniqueName: \"kubernetes.io/projected/048eb183-2cd4-49fc-b6fc-2c4662a29637-kube-api-access-mhmzg\") pod \"048eb183-2cd4-49fc-b6fc-2c4662a29637\" (UID: \"048eb183-2cd4-49fc-b6fc-2c4662a29637\") "
Nov 02 13:30:02.469251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.469222    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-openshift-ca" pod "048eb183-2cd4-49fc-b6fc-2c4662a29637" (UID: "048eb183-2cd4-49fc-b6fc-2c4662a29637"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:02.469786 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.469757    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:02.471236 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.471211    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-xks-ca" pod "048eb183-2cd4-49fc-b6fc-2c4662a29637" (UID: "048eb183-2cd4-49fc-b6fc-2c4662a29637"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:02.471463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.471440    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048eb183-2cd4-49fc-b6fc-2c4662a29637-kube-api-access-mhmzg" pod "048eb183-2cd4-49fc-b6fc-2c4662a29637" (UID: "048eb183-2cd4-49fc-b6fc-2c4662a29637"). InnerVolumeSpecName "kube-api-access-mhmzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:02.571678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.571642    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/048eb183-2cd4-49fc-b6fc-2c4662a29637-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:02.571678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.571671    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhmzg\" (UniqueName: \"kubernetes.io/projected/048eb183-2cd4-49fc-b6fc-2c4662a29637-kube-api-access-mhmzg\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:02.965910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.965870    5879 generic.go:356] "Generic (PLEG): container finished" podID="357161b9-27d9-4b14-81de-365061ce42f6" containerID="de7638bf0747cf9f7b56e7fbac0c9e4bfe3651f6f0a1a69a99c8854e32318681" exitCode=60
Nov 02 13:30:02.966527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.965952    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj" event={"ID":"357161b9-27d9-4b14-81de-365061ce42f6","Type":"ContainerDied","Data":"de7638bf0747cf9f7b56e7fbac0c9e4bfe3651f6f0a1a69a99c8854e32318681"}
Nov 02 13:30:02.966527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.966390    5879 scope.go:122] "RemoveContainer" containerID="de7638bf0747cf9f7b56e7fbac0c9e4bfe3651f6f0a1a69a99c8854e32318681"
Nov 02 13:30:02.968125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.968098    5879 generic.go:356] "Generic (PLEG): container finished" podID="09b683bc-953d-46d1-bc3b-104adddc15f7" containerID="7c2f777ae2ac363ecd1b6abd02b2f27c315c9550bfec7aa868ab4cb866c2c52b" exitCode=60
Nov 02 13:30:02.968282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.968179    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh" event={"ID":"09b683bc-953d-46d1-bc3b-104adddc15f7","Type":"ContainerDied","Data":"7c2f777ae2ac363ecd1b6abd02b2f27c315c9550bfec7aa868ab4cb866c2c52b"}
Nov 02 13:30:02.968483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.968447    5879 scope.go:122] "RemoveContainer" containerID="7c2f777ae2ac363ecd1b6abd02b2f27c315c9550bfec7aa868ab4cb866c2c52b"
Nov 02 13:30:02.970382 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.970361    5879 generic.go:356] "Generic (PLEG): container finished" podID="299f06eb-e0df-4927-9d60-738e88a03192" containerID="70f900c8df16d107d088d1a7610a0b47b849cf9c3589604b4ed6b008d8a617c0" exitCode=60
Nov 02 13:30:02.970479 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.970445    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw" event={"ID":"299f06eb-e0df-4927-9d60-738e88a03192","Type":"ContainerDied","Data":"70f900c8df16d107d088d1a7610a0b47b849cf9c3589604b4ed6b008d8a617c0"}
Nov 02 13:30:02.970675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.970641    5879 scope.go:122] "RemoveContainer" containerID="70f900c8df16d107d088d1a7610a0b47b849cf9c3589604b4ed6b008d8a617c0"
Nov 02 13:30:02.972299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.972282    5879 generic.go:356] "Generic (PLEG): container finished" podID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" containerID="c3f4791db4fe62f843da6a8747a9cd7894fe14bad8d8819c5c371cb30ad6bad0" exitCode=60
Nov 02 13:30:02.972508 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.972489    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb" event={"ID":"3d463a6e-b5f3-407e-8b41-6836c3b709f1","Type":"ContainerDied","Data":"c3f4791db4fe62f843da6a8747a9cd7894fe14bad8d8819c5c371cb30ad6bad0"}
Nov 02 13:30:02.972611 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.972594    5879 scope.go:122] "RemoveContainer" containerID="c3f4791db4fe62f843da6a8747a9cd7894fe14bad8d8819c5c371cb30ad6bad0"
Nov 02 13:30:02.973970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.973942    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"
Nov 02 13:30:02.974103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.973942    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq" event={"ID":"048eb183-2cd4-49fc-b6fc-2c4662a29637","Type":"ContainerDied","Data":"c1fdf10420715967a25d1a7f4dfaca13ebc631fa3502fd6a38c9c2ab02409c91"}
Nov 02 13:30:02.974103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.974065    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1fdf10420715967a25d1a7f4dfaca13ebc631fa3502fd6a38c9c2ab02409c91"
Nov 02 13:30:02.975581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.975557    5879 generic.go:356] "Generic (PLEG): container finished" podID="90197636-9fa4-4217-9aaf-7148e5b1479f" containerID="49a2855b64dc8396fc956cedf1256db090c318e9aab3b3c4976154fe3f5b4d84" exitCode=60
Nov 02 13:30:02.975687 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.975630    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p" event={"ID":"90197636-9fa4-4217-9aaf-7148e5b1479f","Type":"ContainerDied","Data":"49a2855b64dc8396fc956cedf1256db090c318e9aab3b3c4976154fe3f5b4d84"}
Nov 02 13:30:02.975887 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.975868    5879 scope.go:122] "RemoveContainer" containerID="49a2855b64dc8396fc956cedf1256db090c318e9aab3b3c4976154fe3f5b4d84"
Nov 02 13:30:02.977783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.977761    5879 generic.go:356] "Generic (PLEG): container finished" podID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" containerID="31271c058bd237d84e6652c318980d9b20326de66ff3ce8cd2b62a7fa1e89ea5" exitCode=60
Nov 02 13:30:02.977868 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.977839    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k" event={"ID":"4a30bbaa-5aa8-469e-8c1d-1e58a821b310","Type":"ContainerDied","Data":"31271c058bd237d84e6652c318980d9b20326de66ff3ce8cd2b62a7fa1e89ea5"}
Nov 02 13:30:02.978065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.978047    5879 scope.go:122] "RemoveContainer" containerID="31271c058bd237d84e6652c318980d9b20326de66ff3ce8cd2b62a7fa1e89ea5"
Nov 02 13:30:02.983339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.983237    5879 generic.go:356] "Generic (PLEG): container finished" podID="5e755e54-3856-4cab-818e-36e2aca4076d" containerID="5b1375d1cc1c080784365ef8dddae6f89ecbd6ba095b24b1d69ba153ca0772c9" exitCode=60
Nov 02 13:30:02.983408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.983323    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt" event={"ID":"5e755e54-3856-4cab-818e-36e2aca4076d","Type":"ContainerDied","Data":"5b1375d1cc1c080784365ef8dddae6f89ecbd6ba095b24b1d69ba153ca0772c9"}
Nov 02 13:30:02.983704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.983684    5879 scope.go:122] "RemoveContainer" containerID="5b1375d1cc1c080784365ef8dddae6f89ecbd6ba095b24b1d69ba153ca0772c9"
Nov 02 13:30:02.985465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.985439    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n" event={"ID":"74a07f2c-977a-4740-b9fb-56ee22098f68","Type":"ContainerStarted","Data":"54b24d171120e382eb4f9b194b0a15a47cd5d44ced92eb5c9ad1d647b4247666"}
Nov 02 13:30:02.987217 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.987194    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7" event={"ID":"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0","Type":"ContainerStarted","Data":"f14ddb4dff7e077d411ff47eac1236d3cb69207f9eb154548d75100305fb7646"}
Nov 02 13:30:02.989175 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.989149    5879 generic.go:356] "Generic (PLEG): container finished" podID="acd1e90e-1165-4bde-8d01-3d37aee5802e" containerID="46559e9784e740687233c08c35058ee63ac9ccf9965a51f2bfeca272fc76dc97" exitCode=60
Nov 02 13:30:02.989278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.989234    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5" event={"ID":"acd1e90e-1165-4bde-8d01-3d37aee5802e","Type":"ContainerDied","Data":"46559e9784e740687233c08c35058ee63ac9ccf9965a51f2bfeca272fc76dc97"}
Nov 02 13:30:02.989498 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.989480    5879 scope.go:122] "RemoveContainer" containerID="46559e9784e740687233c08c35058ee63ac9ccf9965a51f2bfeca272fc76dc97"
Nov 02 13:30:02.993137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.993069    5879 generic.go:356] "Generic (PLEG): container finished" podID="ebe86781-bd96-4318-8447-767533806fce" containerID="d8a9c814bd4da8d7ba03e99d8523a928ec175891694d6e143aea4c8d9b0869bc" exitCode=60
Nov 02 13:30:02.993364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.993343    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr" event={"ID":"ebe86781-bd96-4318-8447-767533806fce","Type":"ContainerDied","Data":"d8a9c814bd4da8d7ba03e99d8523a928ec175891694d6e143aea4c8d9b0869bc"}
Nov 02 13:30:02.993498 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.993486    5879 scope.go:122] "RemoveContainer" containerID="d8a9c814bd4da8d7ba03e99d8523a928ec175891694d6e143aea4c8d9b0869bc"
Nov 02 13:30:02.996490 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.996466    5879 generic.go:356] "Generic (PLEG): container finished" podID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" containerID="457f525d06628d373aac47937ed7a0d5d4849857313ad2eae9d31e41286f4875" exitCode=60
Nov 02 13:30:02.996588 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.996508    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8" event={"ID":"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136","Type":"ContainerDied","Data":"457f525d06628d373aac47937ed7a0d5d4849857313ad2eae9d31e41286f4875"}
Nov 02 13:30:02.996761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:02.996748    5879 scope.go:122] "RemoveContainer" containerID="457f525d06628d373aac47937ed7a0d5d4849857313ad2eae9d31e41286f4875"
Nov 02 13:30:03.236337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:03.236279    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7" podStartSLOduration=3.236261101 podStartE2EDuration="3.236261101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:03.235447625 +0000 UTC m=+3749.342150480" watchObservedRunningTime="2026-11-02 13:30:03.236261101 +0000 UTC m=+3749.342963956"
Nov 02 13:30:03.521846 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:03.521792    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n" podStartSLOduration=3.521776867 podStartE2EDuration="3.521776867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:03.520553504 +0000 UTC m=+3749.627256356" watchObservedRunningTime="2026-11-02 13:30:03.521776867 +0000 UTC m=+3749.628479739"
Nov 02 13:30:03.805393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:03.805024    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:04.007610 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.007576    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p" event={"ID":"90197636-9fa4-4217-9aaf-7148e5b1479f","Type":"ContainerStarted","Data":"c101c75e8eef10280b78291e3cdf734b2d4c7f6b44b9cae473a7e89efcb8df0f"}
Nov 02 13:30:04.011514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.011483    5879 generic.go:356] "Generic (PLEG): container finished" podID="74a07f2c-977a-4740-b9fb-56ee22098f68" containerID="54b24d171120e382eb4f9b194b0a15a47cd5d44ced92eb5c9ad1d647b4247666" exitCode=60
Nov 02 13:30:04.011770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.011752    5879 scope.go:122] "RemoveContainer" containerID="54b24d171120e382eb4f9b194b0a15a47cd5d44ced92eb5c9ad1d647b4247666"
Nov 02 13:30:04.013475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.013455    5879 generic.go:356] "Generic (PLEG): container finished" podID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" containerID="f14ddb4dff7e077d411ff47eac1236d3cb69207f9eb154548d75100305fb7646" exitCode=60
Nov 02 13:30:04.013817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.013796    5879 scope.go:122] "RemoveContainer" containerID="f14ddb4dff7e077d411ff47eac1236d3cb69207f9eb154548d75100305fb7646"
Nov 02 13:30:04.021405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.021376    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k" event={"ID":"4a30bbaa-5aa8-469e-8c1d-1e58a821b310","Type":"ContainerStarted","Data":"9b186afc8e41adc648bd15fb9c90f2db34fd633b28f9e7c56703a1ec16e2b8c0"}
Nov 02 13:30:04.021496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.021436    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n" event={"ID":"74a07f2c-977a-4740-b9fb-56ee22098f68","Type":"ContainerDied","Data":"54b24d171120e382eb4f9b194b0a15a47cd5d44ced92eb5c9ad1d647b4247666"}
Nov 02 13:30:04.021496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.021462    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7" event={"ID":"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0","Type":"ContainerDied","Data":"f14ddb4dff7e077d411ff47eac1236d3cb69207f9eb154548d75100305fb7646"}
Nov 02 13:30:04.021496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.021485    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj" event={"ID":"357161b9-27d9-4b14-81de-365061ce42f6","Type":"ContainerStarted","Data":"c691bac6df2be15f7a5debee60bc174fda13b3654bcb0f13f66a0150c711e007"}
Nov 02 13:30:04.022766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.022742    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw" event={"ID":"299f06eb-e0df-4927-9d60-738e88a03192","Type":"ContainerStarted","Data":"0f4cd30457088c598df7610e1df202f10696b138d09d2b5e07df4d95db00f3f0"}
Nov 02 13:30:04.056433 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.056374    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p" podStartSLOduration=4.056358621 podStartE2EDuration="4.056358621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:04.053412059 +0000 UTC m=+3750.160114922" watchObservedRunningTime="2026-11-02 13:30:04.056358621 +0000 UTC m=+3750.163061473"
Nov 02 13:30:04.203747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.203696    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw" podStartSLOduration=4.203684254 podStartE2EDuration="4.203684254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:04.200972458 +0000 UTC m=+3750.307675308" watchObservedRunningTime="2026-11-02 13:30:04.203684254 +0000 UTC m=+3750.310387152"
Nov 02 13:30:04.247514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.247455    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8" podStartSLOduration=4.24743548 podStartE2EDuration="4.24743548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:04.246286718 +0000 UTC m=+3750.352989570" watchObservedRunningTime="2026-11-02 13:30:04.24743548 +0000 UTC m=+3750.354138357"
Nov 02 13:30:04.358150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:04.358094    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k" podStartSLOduration=4.358076322 podStartE2EDuration="4.358076322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:04.354960816 +0000 UTC m=+3750.461663666" watchObservedRunningTime="2026-11-02 13:30:04.358076322 +0000 UTC m=+3750.464779227"
Nov 02 13:30:05.032309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.032215    5879 generic.go:356] "Generic (PLEG): container finished" podID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" containerID="14b8be3f989db19eb7a67085f6d955b6d741be02d348fc071965aab0c7acb871" exitCode=60
Nov 02 13:30:05.032309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.032288    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8" event={"ID":"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136","Type":"ContainerDied","Data":"14b8be3f989db19eb7a67085f6d955b6d741be02d348fc071965aab0c7acb871"}
Nov 02 13:30:05.032810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.032354    5879 scope.go:122] "RemoveContainer" containerID="457f525d06628d373aac47937ed7a0d5d4849857313ad2eae9d31e41286f4875"
Nov 02 13:30:05.032810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.032729    5879 scope.go:122] "RemoveContainer" containerID="14b8be3f989db19eb7a67085f6d955b6d741be02d348fc071965aab0c7acb871"
Nov 02 13:30:05.033118 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:05.033093    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8_odh-ai-gateway-infra(bf789a2a-d8c0-4a66-b2dd-d7a1bf968136)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8" podUID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136"
Nov 02 13:30:05.034395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.034377    5879 generic.go:356] "Generic (PLEG): container finished" podID="357161b9-27d9-4b14-81de-365061ce42f6" containerID="c691bac6df2be15f7a5debee60bc174fda13b3654bcb0f13f66a0150c711e007" exitCode=60
Nov 02 13:30:05.034508 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.034431    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj" event={"ID":"357161b9-27d9-4b14-81de-365061ce42f6","Type":"ContainerDied","Data":"c691bac6df2be15f7a5debee60bc174fda13b3654bcb0f13f66a0150c711e007"}
Nov 02 13:30:05.034722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.034707    5879 scope.go:122] "RemoveContainer" containerID="c691bac6df2be15f7a5debee60bc174fda13b3654bcb0f13f66a0150c711e007"
Nov 02 13:30:05.035033 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:05.034982    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj_odh-ai-gateway-infra(357161b9-27d9-4b14-81de-365061ce42f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj" podUID="357161b9-27d9-4b14-81de-365061ce42f6"
Nov 02 13:30:05.036303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.036284    5879 generic.go:356] "Generic (PLEG): container finished" podID="09b683bc-953d-46d1-bc3b-104adddc15f7" containerID="e9f099d227f6ab744ebc6c56fc326b3232d6a1cea6dfd48c0244fc1cfb69a77d" exitCode=60
Nov 02 13:30:05.036399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.036374    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh" event={"ID":"09b683bc-953d-46d1-bc3b-104adddc15f7","Type":"ContainerDied","Data":"e9f099d227f6ab744ebc6c56fc326b3232d6a1cea6dfd48c0244fc1cfb69a77d"}
Nov 02 13:30:05.036680 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.036664    5879 scope.go:122] "RemoveContainer" containerID="e9f099d227f6ab744ebc6c56fc326b3232d6a1cea6dfd48c0244fc1cfb69a77d"
Nov 02 13:30:05.036961 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:05.036938    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh_odh-ai-gateway-infra(09b683bc-953d-46d1-bc3b-104adddc15f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7"
Nov 02 13:30:05.038883 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.038858    5879 generic.go:356] "Generic (PLEG): container finished" podID="299f06eb-e0df-4927-9d60-738e88a03192" containerID="0f4cd30457088c598df7610e1df202f10696b138d09d2b5e07df4d95db00f3f0" exitCode=60
Nov 02 13:30:05.039014 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.038937    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw" event={"ID":"299f06eb-e0df-4927-9d60-738e88a03192","Type":"ContainerDied","Data":"0f4cd30457088c598df7610e1df202f10696b138d09d2b5e07df4d95db00f3f0"}
Nov 02 13:30:05.039225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.039207    5879 scope.go:122] "RemoveContainer" containerID="0f4cd30457088c598df7610e1df202f10696b138d09d2b5e07df4d95db00f3f0"
Nov 02 13:30:05.039517 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:05.039495    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw_odh-ai-gateway-infra(299f06eb-e0df-4927-9d60-738e88a03192)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw" podUID="299f06eb-e0df-4927-9d60-738e88a03192"
Nov 02 13:30:05.041144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.041122    5879 generic.go:356] "Generic (PLEG): container finished" podID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" containerID="c82b1b466d3923d0a4ecbc5747f504f4c0f22f7e9e002639d61dec95aaeb2d15" exitCode=60
Nov 02 13:30:05.041233 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.041201    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb" event={"ID":"3d463a6e-b5f3-407e-8b41-6836c3b709f1","Type":"ContainerDied","Data":"c82b1b466d3923d0a4ecbc5747f504f4c0f22f7e9e002639d61dec95aaeb2d15"}
Nov 02 13:30:05.041426 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.041412    5879 scope.go:122] "RemoveContainer" containerID="c82b1b466d3923d0a4ecbc5747f504f4c0f22f7e9e002639d61dec95aaeb2d15"
Nov 02 13:30:05.041666 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:05.041647    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb_odh-ai-gateway-infra(3d463a6e-b5f3-407e-8b41-6836c3b709f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1"
Nov 02 13:30:05.043160 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.043140    5879 generic.go:356] "Generic (PLEG): container finished" podID="90197636-9fa4-4217-9aaf-7148e5b1479f" containerID="c101c75e8eef10280b78291e3cdf734b2d4c7f6b44b9cae473a7e89efcb8df0f" exitCode=60
Nov 02 13:30:05.043260 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.043215    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p" event={"ID":"90197636-9fa4-4217-9aaf-7148e5b1479f","Type":"ContainerDied","Data":"c101c75e8eef10280b78291e3cdf734b2d4c7f6b44b9cae473a7e89efcb8df0f"}
Nov 02 13:30:05.043431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.043413    5879 scope.go:122] "RemoveContainer" containerID="c101c75e8eef10280b78291e3cdf734b2d4c7f6b44b9cae473a7e89efcb8df0f"
Nov 02 13:30:05.043690 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:05.043663    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p_odh-ai-gateway-infra(90197636-9fa4-4217-9aaf-7148e5b1479f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f"
Nov 02 13:30:05.045220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.045202    5879 generic.go:356] "Generic (PLEG): container finished" podID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" containerID="9b186afc8e41adc648bd15fb9c90f2db34fd633b28f9e7c56703a1ec16e2b8c0" exitCode=60
Nov 02 13:30:05.045316 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.045278    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k" event={"ID":"4a30bbaa-5aa8-469e-8c1d-1e58a821b310","Type":"ContainerDied","Data":"9b186afc8e41adc648bd15fb9c90f2db34fd633b28f9e7c56703a1ec16e2b8c0"}
Nov 02 13:30:05.045489 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.045465    5879 scope.go:122] "RemoveContainer" containerID="9b186afc8e41adc648bd15fb9c90f2db34fd633b28f9e7c56703a1ec16e2b8c0"
Nov 02 13:30:05.045730 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:05.045709    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k_odh-ai-gateway-infra(4a30bbaa-5aa8-469e-8c1d-1e58a821b310)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310"
Nov 02 13:30:05.047190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.047170    5879 generic.go:356] "Generic (PLEG): container finished" podID="5e755e54-3856-4cab-818e-36e2aca4076d" containerID="822d5b13db8e67977d26d7db32bf7a281d97ecffca11142694b3ee6b5d3d95ba" exitCode=60
Nov 02 13:30:05.047277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.047262    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt" event={"ID":"5e755e54-3856-4cab-818e-36e2aca4076d","Type":"ContainerDied","Data":"822d5b13db8e67977d26d7db32bf7a281d97ecffca11142694b3ee6b5d3d95ba"}
Nov 02 13:30:05.047467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.047450    5879 scope.go:122] "RemoveContainer" containerID="822d5b13db8e67977d26d7db32bf7a281d97ecffca11142694b3ee6b5d3d95ba"
Nov 02 13:30:05.047697 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:05.047679    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt_odh-ai-gateway-infra(5e755e54-3856-4cab-818e-36e2aca4076d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt" podUID="5e755e54-3856-4cab-818e-36e2aca4076d"
Nov 02 13:30:05.049419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.049397    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n" event={"ID":"74a07f2c-977a-4740-b9fb-56ee22098f68","Type":"ContainerStarted","Data":"a5b354c5980656e94fd0e4f0f69993029cb928083428de638300127b9e15a0eb"}
Nov 02 13:30:05.051774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.051755    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7" event={"ID":"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0","Type":"ContainerStarted","Data":"3029cf06a1938aa838b695ed2a27ef6b71fadfc6b93c97ec86ad06bad41f1eb1"}
Nov 02 13:30:05.053832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.053810    5879 generic.go:356] "Generic (PLEG): container finished" podID="acd1e90e-1165-4bde-8d01-3d37aee5802e" containerID="e028724dc69171c24b7d44a51e58a954769f6a40bd07a1bc246ccf85079ae5cd" exitCode=60
Nov 02 13:30:05.053926 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.053885    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5" event={"ID":"acd1e90e-1165-4bde-8d01-3d37aee5802e","Type":"ContainerDied","Data":"e028724dc69171c24b7d44a51e58a954769f6a40bd07a1bc246ccf85079ae5cd"}
Nov 02 13:30:05.054177 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.054160    5879 scope.go:122] "RemoveContainer" containerID="e028724dc69171c24b7d44a51e58a954769f6a40bd07a1bc246ccf85079ae5cd"
Nov 02 13:30:05.054424 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:05.054399    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5_odh-ai-gateway-infra(acd1e90e-1165-4bde-8d01-3d37aee5802e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e"
Nov 02 13:30:05.056044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.056024    5879 generic.go:356] "Generic (PLEG): container finished" podID="ebe86781-bd96-4318-8447-767533806fce" containerID="f3b5ead3c96cbb4ef1025a194bcf1c10987c3f02dbe5497634fd7666a8aa7a2e" exitCode=60
Nov 02 13:30:05.056123 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.056067    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr" event={"ID":"ebe86781-bd96-4318-8447-767533806fce","Type":"ContainerDied","Data":"f3b5ead3c96cbb4ef1025a194bcf1c10987c3f02dbe5497634fd7666a8aa7a2e"}
Nov 02 13:30:05.056257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.056243    5879 scope.go:122] "RemoveContainer" containerID="f3b5ead3c96cbb4ef1025a194bcf1c10987c3f02dbe5497634fd7666a8aa7a2e"
Nov 02 13:30:05.056462 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:05.056441    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr_odh-ai-gateway-infra(ebe86781-bd96-4318-8447-767533806fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr" podUID="ebe86781-bd96-4318-8447-767533806fce"
Nov 02 13:30:05.063880 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.063860    5879 scope.go:122] "RemoveContainer" containerID="de7638bf0747cf9f7b56e7fbac0c9e4bfe3651f6f0a1a69a99c8854e32318681"
Nov 02 13:30:05.078289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.078269    5879 scope.go:122] "RemoveContainer" containerID="7c2f777ae2ac363ecd1b6abd02b2f27c315c9550bfec7aa868ab4cb866c2c52b"
Nov 02 13:30:05.094351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.094326    5879 scope.go:122] "RemoveContainer" containerID="70f900c8df16d107d088d1a7610a0b47b849cf9c3589604b4ed6b008d8a617c0"
Nov 02 13:30:05.108884 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.108865    5879 scope.go:122] "RemoveContainer" containerID="c3f4791db4fe62f843da6a8747a9cd7894fe14bad8d8819c5c371cb30ad6bad0"
Nov 02 13:30:05.125010 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.124969    5879 scope.go:122] "RemoveContainer" containerID="49a2855b64dc8396fc956cedf1256db090c318e9aab3b3c4976154fe3f5b4d84"
Nov 02 13:30:05.140205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.140182    5879 scope.go:122] "RemoveContainer" containerID="31271c058bd237d84e6652c318980d9b20326de66ff3ce8cd2b62a7fa1e89ea5"
Nov 02 13:30:05.155748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.155720    5879 scope.go:122] "RemoveContainer" containerID="5b1375d1cc1c080784365ef8dddae6f89ecbd6ba095b24b1d69ba153ca0772c9"
Nov 02 13:30:05.170462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.170442    5879 scope.go:122] "RemoveContainer" containerID="46559e9784e740687233c08c35058ee63ac9ccf9965a51f2bfeca272fc76dc97"
Nov 02 13:30:05.187724 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.187695    5879 scope.go:122] "RemoveContainer" containerID="d8a9c814bd4da8d7ba03e99d8523a928ec175891694d6e143aea4c8d9b0869bc"
Nov 02 13:30:05.503238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.503199    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"]
Nov 02 13:30:05.508726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:05.508683    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:06.066360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.066328    5879 scope.go:122] "RemoveContainer" containerID="c691bac6df2be15f7a5debee60bc174fda13b3654bcb0f13f66a0150c711e007"
Nov 02 13:30:06.066822 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:06.066747    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj_odh-ai-gateway-infra(357161b9-27d9-4b14-81de-365061ce42f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj" podUID="357161b9-27d9-4b14-81de-365061ce42f6"
Nov 02 13:30:06.068086 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.068052    5879 scope.go:122] "RemoveContainer" containerID="e9f099d227f6ab744ebc6c56fc326b3232d6a1cea6dfd48c0244fc1cfb69a77d"
Nov 02 13:30:06.068367 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:06.068342    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh_odh-ai-gateway-infra(09b683bc-953d-46d1-bc3b-104adddc15f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7"
Nov 02 13:30:06.069718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.069699    5879 scope.go:122] "RemoveContainer" containerID="0f4cd30457088c598df7610e1df202f10696b138d09d2b5e07df4d95db00f3f0"
Nov 02 13:30:06.070049 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:06.069978    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw_odh-ai-gateway-infra(299f06eb-e0df-4927-9d60-738e88a03192)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw" podUID="299f06eb-e0df-4927-9d60-738e88a03192"
Nov 02 13:30:06.071568 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.071539    5879 scope.go:122] "RemoveContainer" containerID="c82b1b466d3923d0a4ecbc5747f504f4c0f22f7e9e002639d61dec95aaeb2d15"
Nov 02 13:30:06.071845 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:06.071823    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb_odh-ai-gateway-infra(3d463a6e-b5f3-407e-8b41-6836c3b709f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1"
Nov 02 13:30:06.073146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.073129    5879 scope.go:122] "RemoveContainer" containerID="c101c75e8eef10280b78291e3cdf734b2d4c7f6b44b9cae473a7e89efcb8df0f"
Nov 02 13:30:06.073410 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:06.073389    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p_odh-ai-gateway-infra(90197636-9fa4-4217-9aaf-7148e5b1479f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f"
Nov 02 13:30:06.075285 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.075268    5879 scope.go:122] "RemoveContainer" containerID="9b186afc8e41adc648bd15fb9c90f2db34fd633b28f9e7c56703a1ec16e2b8c0"
Nov 02 13:30:06.075530 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:06.075511    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k_odh-ai-gateway-infra(4a30bbaa-5aa8-469e-8c1d-1e58a821b310)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310"
Nov 02 13:30:06.077095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.077072    5879 scope.go:122] "RemoveContainer" containerID="822d5b13db8e67977d26d7db32bf7a281d97ecffca11142694b3ee6b5d3d95ba"
Nov 02 13:30:06.077360 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:06.077340    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt_odh-ai-gateway-infra(5e755e54-3856-4cab-818e-36e2aca4076d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt" podUID="5e755e54-3856-4cab-818e-36e2aca4076d"
Nov 02 13:30:06.078966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.078944    5879 generic.go:356] "Generic (PLEG): container finished" podID="74a07f2c-977a-4740-b9fb-56ee22098f68" containerID="a5b354c5980656e94fd0e4f0f69993029cb928083428de638300127b9e15a0eb" exitCode=60
Nov 02 13:30:06.079105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.079026    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n" event={"ID":"74a07f2c-977a-4740-b9fb-56ee22098f68","Type":"ContainerDied","Data":"a5b354c5980656e94fd0e4f0f69993029cb928083428de638300127b9e15a0eb"}
Nov 02 13:30:06.079105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.079088    5879 scope.go:122] "RemoveContainer" containerID="54b24d171120e382eb4f9b194b0a15a47cd5d44ced92eb5c9ad1d647b4247666"
Nov 02 13:30:06.079283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.079269    5879 scope.go:122] "RemoveContainer" containerID="a5b354c5980656e94fd0e4f0f69993029cb928083428de638300127b9e15a0eb"
Nov 02 13:30:06.079524 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:06.079505    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n_odh-ai-gateway-infra(74a07f2c-977a-4740-b9fb-56ee22098f68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n" podUID="74a07f2c-977a-4740-b9fb-56ee22098f68"
Nov 02 13:30:06.080958 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.080935    5879 generic.go:356] "Generic (PLEG): container finished" podID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" containerID="3029cf06a1938aa838b695ed2a27ef6b71fadfc6b93c97ec86ad06bad41f1eb1" exitCode=60
Nov 02 13:30:06.081081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.080958    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7" event={"ID":"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0","Type":"ContainerDied","Data":"3029cf06a1938aa838b695ed2a27ef6b71fadfc6b93c97ec86ad06bad41f1eb1"}
Nov 02 13:30:06.081244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.081230    5879 scope.go:122] "RemoveContainer" containerID="3029cf06a1938aa838b695ed2a27ef6b71fadfc6b93c97ec86ad06bad41f1eb1"
Nov 02 13:30:06.081489 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:06.081464    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7_odh-ai-gateway-infra(5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0"
Nov 02 13:30:06.082850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.082834    5879 scope.go:122] "RemoveContainer" containerID="e028724dc69171c24b7d44a51e58a954769f6a40bd07a1bc246ccf85079ae5cd"
Nov 02 13:30:06.083088 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:06.083064    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5_odh-ai-gateway-infra(acd1e90e-1165-4bde-8d01-3d37aee5802e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e"
Nov 02 13:30:06.084656 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.084640    5879 scope.go:122] "RemoveContainer" containerID="f3b5ead3c96cbb4ef1025a194bcf1c10987c3f02dbe5497634fd7666a8aa7a2e"
Nov 02 13:30:06.084868 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:06.084846    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr_odh-ai-gateway-infra(ebe86781-bd96-4318-8447-767533806fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr" podUID="ebe86781-bd96-4318-8447-767533806fce"
Nov 02 13:30:06.093655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.093636    5879 scope.go:122] "RemoveContainer" containerID="f14ddb4dff7e077d411ff47eac1236d3cb69207f9eb154548d75100305fb7646"
Nov 02 13:30:06.198580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.198533    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:06.859531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.859493    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main" containerID="cri-o://e7613b05a9a0d8f1e65e6f20cca3df8c9f0f8eb728d1c124cec4843a64b423f0" gracePeriod=45
Nov 02 13:30:06.871067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.871036    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" containerID="cri-o://89d66747eee021b156db4c9493412727e1b4b8cd138ae79c08212bf90c8fc42b" gracePeriod=45
Nov 02 13:30:06.910074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:06.910022    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": dial tcp 10.134.0.48:8000: connect: connection refused"
Nov 02 13:30:07.103025 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.100784    5879 generic.go:356] "Generic (PLEG): container finished" podID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerID="e7613b05a9a0d8f1e65e6f20cca3df8c9f0f8eb728d1c124cec4843a64b423f0" exitCode=0
Nov 02 13:30:07.103025 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.101014    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" event={"ID":"03ad3fad-c135-4470-9d9d-429d0d2df2b4","Type":"ContainerDied","Data":"e7613b05a9a0d8f1e65e6f20cca3df8c9f0f8eb728d1c124cec4843a64b423f0"}
Nov 02 13:30:07.105416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.104403    5879 scope.go:122] "RemoveContainer" containerID="a5b354c5980656e94fd0e4f0f69993029cb928083428de638300127b9e15a0eb"
Nov 02 13:30:07.105416 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:07.104799    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n_odh-ai-gateway-infra(74a07f2c-977a-4740-b9fb-56ee22098f68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n" podUID="74a07f2c-977a-4740-b9fb-56ee22098f68"
Nov 02 13:30:07.108440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.108032    5879 scope.go:122] "RemoveContainer" containerID="3029cf06a1938aa838b695ed2a27ef6b71fadfc6b93c97ec86ad06bad41f1eb1"
Nov 02 13:30:07.108440 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:07.108369    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7_odh-ai-gateway-infra(5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0"
Nov 02 13:30:07.110895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.110660    5879 generic.go:356] "Generic (PLEG): container finished" podID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerID="89d66747eee021b156db4c9493412727e1b4b8cd138ae79c08212bf90c8fc42b" exitCode=0
Nov 02 13:30:07.110895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.110811    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" event={"ID":"ac7edc2c-4ae5-4714-a9b6-1b07c6700083","Type":"ContainerDied","Data":"89d66747eee021b156db4c9493412727e1b4b8cd138ae79c08212bf90c8fc42b"}
Nov 02 13:30:07.199615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.196881    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:07.285260 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.285232    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 13:30:07.287093 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.287066    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 13:30:07.346826 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.346798    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tls-certs\") pod \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") "
Nov 02 13:30:07.347045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.346836    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-model-cache\") pod \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") "
Nov 02 13:30:07.347045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.346857    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-home\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-home\") pod \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") "
Nov 02 13:30:07.347045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.346978    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ad3fad-c135-4470-9d9d-429d0d2df2b4-kube-api-access-z5dlj\" (UniqueName: \"kubernetes.io/projected/03ad3fad-c135-4470-9d9d-429d0d2df2b4-kube-api-access-z5dlj\") pod \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") "
Nov 02 13:30:07.347336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.347055    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"
Nov 02 13:30:07.347336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.347139    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-dshm\") pod \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") "
Nov 02 13:30:07.347336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.347160    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-model-cache" pod "ac7edc2c-4ae5-4714-a9b6-1b07c6700083" (UID: "ac7edc2c-4ae5-4714-a9b6-1b07c6700083"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.347336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.347208    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-kube-api-access-9mbzk\" (UniqueName: \"kubernetes.io/projected/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-kube-api-access-9mbzk\") pod \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") "
Nov 02 13:30:07.347336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.347260    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tls-certs\") pod \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") "
Nov 02 13:30:07.347336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.347292    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-model-cache\") pod \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") "
Nov 02 13:30:07.347839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.347396    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tmp-dir\") pod \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\" (UID: \"ac7edc2c-4ae5-4714-a9b6-1b07c6700083\") "
Nov 02 13:30:07.347839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.347466    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-home\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-home\") pod \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") "
Nov 02 13:30:07.347839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.347505    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-dshm\") pod \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") "
Nov 02 13:30:07.347839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.347546    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tmp-dir\") pod \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\" (UID: \"03ad3fad-c135-4470-9d9d-429d0d2df2b4\") "
Nov 02 13:30:07.347839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.347634    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-model-cache" pod "03ad3fad-c135-4470-9d9d-429d0d2df2b4" (UID: "03ad3fad-c135-4470-9d9d-429d0d2df2b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.348149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.348045    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-home" pod "03ad3fad-c135-4470-9d9d-429d0d2df2b4" (UID: "03ad3fad-c135-4470-9d9d-429d0d2df2b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.348149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.348096    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-home" pod "ac7edc2c-4ae5-4714-a9b6-1b07c6700083" (UID: "ac7edc2c-4ae5-4714-a9b6-1b07c6700083"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.348149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.348114    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tmp-dir" pod "ac7edc2c-4ae5-4714-a9b6-1b07c6700083" (UID: "ac7edc2c-4ae5-4714-a9b6-1b07c6700083"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.348336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.348267    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tmp-dir" pod "03ad3fad-c135-4470-9d9d-429d0d2df2b4" (UID: "03ad3fad-c135-4470-9d9d-429d0d2df2b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.348437 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.348418    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.348500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.348443    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.348500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.348455    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.348500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.348466    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.348500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.348478    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.348500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.348489    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.350299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.350263    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tls-certs" pod "03ad3fad-c135-4470-9d9d-429d0d2df2b4" (UID: "03ad3fad-c135-4470-9d9d-429d0d2df2b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:07.350716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.350603    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tls-certs" pod "ac7edc2c-4ae5-4714-a9b6-1b07c6700083" (UID: "ac7edc2c-4ae5-4714-a9b6-1b07c6700083"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:07.350716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.350696    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ad3fad-c135-4470-9d9d-429d0d2df2b4-kube-api-access-z5dlj" pod "03ad3fad-c135-4470-9d9d-429d0d2df2b4" (UID: "03ad3fad-c135-4470-9d9d-429d0d2df2b4"). InnerVolumeSpecName "kube-api-access-z5dlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:07.350886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.350866    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-dshm" pod "03ad3fad-c135-4470-9d9d-429d0d2df2b4" (UID: "03ad3fad-c135-4470-9d9d-429d0d2df2b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.351558 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.351539    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-kube-api-access-9mbzk" pod "ac7edc2c-4ae5-4714-a9b6-1b07c6700083" (UID: "ac7edc2c-4ae5-4714-a9b6-1b07c6700083"). InnerVolumeSpecName "kube-api-access-9mbzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:07.352158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.352139    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-dshm" pod "ac7edc2c-4ae5-4714-a9b6-1b07c6700083" (UID: "ac7edc2c-4ae5-4714-a9b6-1b07c6700083"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:30:07.355138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.355117    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:07.452075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.452043    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e755e54-3856-4cab-818e-36e2aca4076d-kube-api-access-pzcrb\" (UniqueName: \"kubernetes.io/projected/5e755e54-3856-4cab-818e-36e2aca4076d-kube-api-access-pzcrb\") pod \"5e755e54-3856-4cab-818e-36e2aca4076d\" (UID: \"5e755e54-3856-4cab-818e-36e2aca4076d\") "
Nov 02 13:30:07.452373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.452348    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e755e54-3856-4cab-818e-36e2aca4076d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e755e54-3856-4cab-818e-36e2aca4076d-maas-api-service-ca\") pod \"5e755e54-3856-4cab-818e-36e2aca4076d\" (UID: \"5e755e54-3856-4cab-818e-36e2aca4076d\") "
Nov 02 13:30:07.453237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.453072    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03ad3fad-c135-4470-9d9d-429d0d2df2b4-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.453237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.453099    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.453237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.453113    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5dlj\" (UniqueName: \"kubernetes.io/projected/03ad3fad-c135-4470-9d9d-429d0d2df2b4-kube-api-access-z5dlj\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.453237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.453125    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.453237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.453138    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mbzk\" (UniqueName: \"kubernetes.io/projected/ac7edc2c-4ae5-4714-a9b6-1b07c6700083-kube-api-access-9mbzk\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.453237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.453150    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03ad3fad-c135-4470-9d9d-429d0d2df2b4-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.453897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.453506    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e755e54-3856-4cab-818e-36e2aca4076d-maas-api-service-ca" pod "5e755e54-3856-4cab-818e-36e2aca4076d" (UID: "5e755e54-3856-4cab-818e-36e2aca4076d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:07.457054 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.457027    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e755e54-3856-4cab-818e-36e2aca4076d-kube-api-access-pzcrb" pod "5e755e54-3856-4cab-818e-36e2aca4076d" (UID: "5e755e54-3856-4cab-818e-36e2aca4076d"). InnerVolumeSpecName "kube-api-access-pzcrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:07.555088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.555059    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e755e54-3856-4cab-818e-36e2aca4076d-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.555088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.555086    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzcrb\" (UniqueName: \"kubernetes.io/projected/5e755e54-3856-4cab-818e-36e2aca4076d-kube-api-access-pzcrb\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:07.938940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:07.938906    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:08.120270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.120232    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"
Nov 02 13:30:08.120803 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.120237    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8" event={"ID":"ac7edc2c-4ae5-4714-a9b6-1b07c6700083","Type":"ContainerDied","Data":"d231650d012a85ebde453bb8c8b3a5d5c82e1da9838eae61998c30148d1da586"}
Nov 02 13:30:08.120803 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.120371    5879 scope.go:122] "RemoveContainer" containerID="89d66747eee021b156db4c9493412727e1b4b8cd138ae79c08212bf90c8fc42b"
Nov 02 13:30:08.121935 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.121916    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"
Nov 02 13:30:08.122088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.121947    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n" event={"ID":"03ad3fad-c135-4470-9d9d-429d0d2df2b4","Type":"ContainerDied","Data":"9386e29414c22c2ad76ba3a56e6ea5470b9baadec896b9a0ca57a947d9b6f8dd"}
Nov 02 13:30:08.123765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.123740    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt" event={"ID":"5e755e54-3856-4cab-818e-36e2aca4076d","Type":"ContainerDied","Data":"db27f744a57a916671b64f50a9555f7421ad1a34ad3ff41cc58b81be0f4d6416"}
Nov 02 13:30:08.123765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.123761    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"
Nov 02 13:30:08.132711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.132694    5879 scope.go:122] "RemoveContainer" containerID="e7613b05a9a0d8f1e65e6f20cca3df8c9f0f8eb728d1c124cec4843a64b423f0"
Nov 02 13:30:08.142552 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.142533    5879 scope.go:122] "RemoveContainer" containerID="822d5b13db8e67977d26d7db32bf7a281d97ecffca11142694b3ee6b5d3d95ba"
Nov 02 13:30:08.179124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.179084    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:08.197654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.197620    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"]
Nov 02 13:30:08.205980 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.205949    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-00d004-29893770-9sjxt"]
Nov 02 13:30:08.254473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.254437    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"]
Nov 02 13:30:08.260669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.260633    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-843978/premium-w4-843978-kserve-58f57bc4c9-d7n7n"]
Nov 02 13:30:08.313980 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.313942    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"]
Nov 02 13:30:08.317652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.317623    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-843978/simulator-w4-843978-kserve-748d75d666-qzxl8"]
Nov 02 13:30:08.333683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:08.333655    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:09.122866 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:09.122828    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:09.246187 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:09.246087    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:09.955348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:09.955318    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:10.019586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:10.019551    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" path="/var/lib/kubelet/pods/03ad3fad-c135-4470-9d9d-429d0d2df2b4/volumes"
Nov 02 13:30:10.020094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:10.020079    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e755e54-3856-4cab-818e-36e2aca4076d" path="/var/lib/kubelet/pods/5e755e54-3856-4cab-818e-36e2aca4076d/volumes"
Nov 02 13:30:10.020557 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:10.020543    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" path="/var/lib/kubelet/pods/ac7edc2c-4ae5-4714-a9b6-1b07c6700083/volumes"
Nov 02 13:30:11.712747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:11.712711    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:13.451524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:13.451459    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:14.828884 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:14.828848    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:30:14.854383 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:14.854349    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"]
Nov 02 13:30:14.860669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:14.860639    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:15.068829 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.068799    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"
Nov 02 13:30:15.183018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.182573    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a07f2c-977a-4740-b9fb-56ee22098f68-kube-api-access-5k8h2\" (UniqueName: \"kubernetes.io/projected/74a07f2c-977a-4740-b9fb-56ee22098f68-kube-api-access-5k8h2\") pod \"74a07f2c-977a-4740-b9fb-56ee22098f68\" (UID: \"74a07f2c-977a-4740-b9fb-56ee22098f68\") "
Nov 02 13:30:15.183018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.182804    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a07f2c-977a-4740-b9fb-56ee22098f68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a07f2c-977a-4740-b9fb-56ee22098f68-maas-api-service-ca\") pod \"74a07f2c-977a-4740-b9fb-56ee22098f68\" (UID: \"74a07f2c-977a-4740-b9fb-56ee22098f68\") "
Nov 02 13:30:15.200026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.196919    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a07f2c-977a-4740-b9fb-56ee22098f68-maas-api-service-ca" pod "74a07f2c-977a-4740-b9fb-56ee22098f68" (UID: "74a07f2c-977a-4740-b9fb-56ee22098f68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:15.228877 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.227703    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a07f2c-977a-4740-b9fb-56ee22098f68-kube-api-access-5k8h2" pod "74a07f2c-977a-4740-b9fb-56ee22098f68" (UID: "74a07f2c-977a-4740-b9fb-56ee22098f68"). InnerVolumeSpecName "kube-api-access-5k8h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:15.233653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.233611    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n" event={"ID":"74a07f2c-977a-4740-b9fb-56ee22098f68","Type":"ContainerDied","Data":"1038d06905b36cb172a337a6159a4083af09d8900e99f2918297e6e2223078ef"}
Nov 02 13:30:15.233833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.233682    5879 scope.go:122] "RemoveContainer" containerID="a5b354c5980656e94fd0e4f0f69993029cb928083428de638300127b9e15a0eb"
Nov 02 13:30:15.233833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.233826    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"
Nov 02 13:30:15.285032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.284983    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74a07f2c-977a-4740-b9fb-56ee22098f68-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:15.285032 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.285025    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k8h2\" (UniqueName: \"kubernetes.io/projected/74a07f2c-977a-4740-b9fb-56ee22098f68-kube-api-access-5k8h2\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:15.310500 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.310469    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"]
Nov 02 13:30:15.316490 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.316463    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-843978-29893770-tq66n"]
Nov 02 13:30:15.543073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:15.542960    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:16.018654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:16.018622    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a07f2c-977a-4740-b9fb-56ee22098f68" path="/var/lib/kubelet/pods/74a07f2c-977a-4740-b9fb-56ee22098f68/volumes"
Nov 02 13:30:16.925576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:16.925534    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:17.020648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:17.016936    5879 scope.go:122] "RemoveContainer" containerID="c691bac6df2be15f7a5debee60bc174fda13b3654bcb0f13f66a0150c711e007"
Nov 02 13:30:17.020648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:17.017626    5879 scope.go:122] "RemoveContainer" containerID="c101c75e8eef10280b78291e3cdf734b2d4c7f6b44b9cae473a7e89efcb8df0f"
Nov 02 13:30:17.020648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:17.018130    5879 scope.go:122] "RemoveContainer" containerID="e028724dc69171c24b7d44a51e58a954769f6a40bd07a1bc246ccf85079ae5cd"
Nov 02 13:30:17.020648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:17.018188    5879 scope.go:122] "RemoveContainer" containerID="3029cf06a1938aa838b695ed2a27ef6b71fadfc6b93c97ec86ad06bad41f1eb1"
Nov 02 13:30:17.020648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:17.018507    5879 scope.go:122] "RemoveContainer" containerID="c82b1b466d3923d0a4ecbc5747f504f4c0f22f7e9e002639d61dec95aaeb2d15"
Nov 02 13:30:17.071242 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:17.071150    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:17.374108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:17.373854    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:17.580216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:17.580123    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:17.735326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:17.735276    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:18.010013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.009966    5879 scope.go:122] "RemoveContainer" containerID="9b186afc8e41adc648bd15fb9c90f2db34fd633b28f9e7c56703a1ec16e2b8c0"
Nov 02 13:30:18.010248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.010206    5879 scope.go:122] "RemoveContainer" containerID="14b8be3f989db19eb7a67085f6d955b6d741be02d348fc071965aab0c7acb871"
Nov 02 13:30:18.010484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.010463    5879 scope.go:122] "RemoveContainer" containerID="0f4cd30457088c598df7610e1df202f10696b138d09d2b5e07df4d95db00f3f0"
Nov 02 13:30:18.010577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.010537    5879 scope.go:122] "RemoveContainer" containerID="e9f099d227f6ab744ebc6c56fc326b3232d6a1cea6dfd48c0244fc1cfb69a77d"
Nov 02 13:30:18.269586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.269496    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj" event={"ID":"357161b9-27d9-4b14-81de-365061ce42f6","Type":"ContainerStarted","Data":"734390a046e2ca2ec2d1e7cddfba2c0a30b6d1e0c6a76a164cdf906b5e46d92d"}
Nov 02 13:30:18.272028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.271983    5879 generic.go:356] "Generic (PLEG): container finished" podID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" containerID="2353a856c4950f94b8c3afd08ec5424b9aa0b21202c7479c36eafc01d1692ac6" exitCode=60
Nov 02 13:30:18.272157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.272045    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb" event={"ID":"3d463a6e-b5f3-407e-8b41-6836c3b709f1","Type":"ContainerDied","Data":"2353a856c4950f94b8c3afd08ec5424b9aa0b21202c7479c36eafc01d1692ac6"}
Nov 02 13:30:18.272157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.272097    5879 scope.go:122] "RemoveContainer" containerID="c82b1b466d3923d0a4ecbc5747f504f4c0f22f7e9e002639d61dec95aaeb2d15"
Nov 02 13:30:18.272551 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.272530    5879 scope.go:122] "RemoveContainer" containerID="2353a856c4950f94b8c3afd08ec5424b9aa0b21202c7479c36eafc01d1692ac6"
Nov 02 13:30:18.272957 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:18.272921    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb_odh-ai-gateway-infra(3d463a6e-b5f3-407e-8b41-6836c3b709f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1"
Nov 02 13:30:18.275115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.275088    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p" event={"ID":"90197636-9fa4-4217-9aaf-7148e5b1479f","Type":"ContainerStarted","Data":"9c784499b10edab0a5bdd4233b2159b8a3a97990b0a77d7d90f77c06b7e03450"}
Nov 02 13:30:18.277366 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.277341    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7" event={"ID":"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0","Type":"ContainerStarted","Data":"d8deb7f74ef3ee20a97638d9840811400e92a457ea7ece8557f285d3545a537d"}
Nov 02 13:30:18.279513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.279494    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5" event={"ID":"acd1e90e-1165-4bde-8d01-3d37aee5802e","Type":"ContainerStarted","Data":"a5ea2b9ee1c0a29664f491327bf6ff0dd96d09776740daf7ad42bbbaaa00a550"}
Nov 02 13:30:18.520119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:18.520008    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5" podStartSLOduration=18.519974656 podStartE2EDuration="18.519974656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:18.517524582 +0000 UTC m=+3764.624227433" watchObservedRunningTime="2026-11-02 13:30:18.519974656 +0000 UTC m=+3764.626677554"
Nov 02 13:30:19.009359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.009325    5879 scope.go:122] "RemoveContainer" containerID="f3b5ead3c96cbb4ef1025a194bcf1c10987c3f02dbe5497634fd7666a8aa7a2e"
Nov 02 13:30:19.298886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.298793    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh" event={"ID":"09b683bc-953d-46d1-bc3b-104adddc15f7","Type":"ContainerStarted","Data":"4500171ed6067a11027cbf403d5b09ab6ad41da34a8a8b869b736fb3a2fb4bbf"}
Nov 02 13:30:19.303355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.303328    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw" event={"ID":"299f06eb-e0df-4927-9d60-738e88a03192","Type":"ContainerStarted","Data":"efd27ae4de97ac42e9f879e1493b485ed5d85daa63d500e98ee49f6249571511"}
Nov 02 13:30:19.307178 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.307154    5879 generic.go:356] "Generic (PLEG): container finished" podID="90197636-9fa4-4217-9aaf-7148e5b1479f" containerID="9c784499b10edab0a5bdd4233b2159b8a3a97990b0a77d7d90f77c06b7e03450" exitCode=60
Nov 02 13:30:19.307304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.307222    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p" event={"ID":"90197636-9fa4-4217-9aaf-7148e5b1479f","Type":"ContainerDied","Data":"9c784499b10edab0a5bdd4233b2159b8a3a97990b0a77d7d90f77c06b7e03450"}
Nov 02 13:30:19.307304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.307269    5879 scope.go:122] "RemoveContainer" containerID="c101c75e8eef10280b78291e3cdf734b2d4c7f6b44b9cae473a7e89efcb8df0f"
Nov 02 13:30:19.307602 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.307584    5879 scope.go:122] "RemoveContainer" containerID="9c784499b10edab0a5bdd4233b2159b8a3a97990b0a77d7d90f77c06b7e03450"
Nov 02 13:30:19.307887 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:19.307864    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p_odh-ai-gateway-infra(90197636-9fa4-4217-9aaf-7148e5b1479f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f"
Nov 02 13:30:19.309191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.309168    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k" event={"ID":"4a30bbaa-5aa8-469e-8c1d-1e58a821b310","Type":"ContainerStarted","Data":"3214eaa966ea7a41b8fea44ad25cb745405e11d53f3a08caea9adc1947178517"}
Nov 02 13:30:19.310903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.310883    5879 generic.go:356] "Generic (PLEG): container finished" podID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" containerID="d8deb7f74ef3ee20a97638d9840811400e92a457ea7ece8557f285d3545a537d" exitCode=60
Nov 02 13:30:19.310986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.310936    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7" event={"ID":"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0","Type":"ContainerDied","Data":"d8deb7f74ef3ee20a97638d9840811400e92a457ea7ece8557f285d3545a537d"}
Nov 02 13:30:19.311266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.311251    5879 scope.go:122] "RemoveContainer" containerID="d8deb7f74ef3ee20a97638d9840811400e92a457ea7ece8557f285d3545a537d"
Nov 02 13:30:19.311469 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:19.311447    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7_odh-ai-gateway-infra(5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0"
Nov 02 13:30:19.312710 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.312679    5879 generic.go:356] "Generic (PLEG): container finished" podID="acd1e90e-1165-4bde-8d01-3d37aee5802e" containerID="a5ea2b9ee1c0a29664f491327bf6ff0dd96d09776740daf7ad42bbbaaa00a550" exitCode=60
Nov 02 13:30:19.312809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.312745    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5" event={"ID":"acd1e90e-1165-4bde-8d01-3d37aee5802e","Type":"ContainerDied","Data":"a5ea2b9ee1c0a29664f491327bf6ff0dd96d09776740daf7ad42bbbaaa00a550"}
Nov 02 13:30:19.313028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.313011    5879 scope.go:122] "RemoveContainer" containerID="a5ea2b9ee1c0a29664f491327bf6ff0dd96d09776740daf7ad42bbbaaa00a550"
Nov 02 13:30:19.313224 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:19.313199    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5_odh-ai-gateway-infra(acd1e90e-1165-4bde-8d01-3d37aee5802e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e"
Nov 02 13:30:19.315028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.314981    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8" event={"ID":"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136","Type":"ContainerStarted","Data":"1924969280e85fc5345b5bc8ffbe65dc466bd14976c1d0c144323ac82707556f"}
Nov 02 13:30:19.316951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.316929    5879 generic.go:356] "Generic (PLEG): container finished" podID="357161b9-27d9-4b14-81de-365061ce42f6" containerID="734390a046e2ca2ec2d1e7cddfba2c0a30b6d1e0c6a76a164cdf906b5e46d92d" exitCode=60
Nov 02 13:30:19.317069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.316986    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj" event={"ID":"357161b9-27d9-4b14-81de-365061ce42f6","Type":"ContainerDied","Data":"734390a046e2ca2ec2d1e7cddfba2c0a30b6d1e0c6a76a164cdf906b5e46d92d"}
Nov 02 13:30:19.317279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.317265    5879 scope.go:122] "RemoveContainer" containerID="734390a046e2ca2ec2d1e7cddfba2c0a30b6d1e0c6a76a164cdf906b5e46d92d"
Nov 02 13:30:19.317465 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:19.317447    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj_odh-ai-gateway-infra(357161b9-27d9-4b14-81de-365061ce42f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj" podUID="357161b9-27d9-4b14-81de-365061ce42f6"
Nov 02 13:30:19.323436 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.323409    5879 scope.go:122] "RemoveContainer" containerID="3029cf06a1938aa838b695ed2a27ef6b71fadfc6b93c97ec86ad06bad41f1eb1"
Nov 02 13:30:19.337555 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.337538    5879 scope.go:122] "RemoveContainer" containerID="e028724dc69171c24b7d44a51e58a954769f6a40bd07a1bc246ccf85079ae5cd"
Nov 02 13:30:19.349582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.349522    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh" podStartSLOduration=19.349508627 podStartE2EDuration="19.349508627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:19.348962388 +0000 UTC m=+3765.455665241" watchObservedRunningTime="2026-11-02 13:30:19.349508627 +0000 UTC m=+3765.456211531"
Nov 02 13:30:19.352254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.352235    5879 scope.go:122] "RemoveContainer" containerID="c691bac6df2be15f7a5debee60bc174fda13b3654bcb0f13f66a0150c711e007"
Nov 02 13:30:19.369101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.369072    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"]
Nov 02 13:30:19.443454 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.443424    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"]
Nov 02 13:30:19.496078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.488082    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"]
Nov 02 13:30:19.506523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.506452    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:19.529671 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.529642    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"]
Nov 02 13:30:19.576213 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.576135    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"]
Nov 02 13:30:19.731374 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.731349    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"
Nov 02 13:30:19.762640 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.762606    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d463a6e-b5f3-407e-8b41-6836c3b709f1-kube-api-access-kd4vz\" (UniqueName: \"kubernetes.io/projected/3d463a6e-b5f3-407e-8b41-6836c3b709f1-kube-api-access-kd4vz\") pod \"3d463a6e-b5f3-407e-8b41-6836c3b709f1\" (UID: \"3d463a6e-b5f3-407e-8b41-6836c3b709f1\") "
Nov 02 13:30:19.762831 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.762785    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d463a6e-b5f3-407e-8b41-6836c3b709f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d463a6e-b5f3-407e-8b41-6836c3b709f1-maas-api-service-ca\") pod \"3d463a6e-b5f3-407e-8b41-6836c3b709f1\" (UID: \"3d463a6e-b5f3-407e-8b41-6836c3b709f1\") "
Nov 02 13:30:19.763191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.763168    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d463a6e-b5f3-407e-8b41-6836c3b709f1-maas-api-service-ca" pod "3d463a6e-b5f3-407e-8b41-6836c3b709f1" (UID: "3d463a6e-b5f3-407e-8b41-6836c3b709f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:19.763474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.763294    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d463a6e-b5f3-407e-8b41-6836c3b709f1-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:19.764918 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.764896    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d463a6e-b5f3-407e-8b41-6836c3b709f1-kube-api-access-kd4vz" pod "3d463a6e-b5f3-407e-8b41-6836c3b709f1" (UID: "3d463a6e-b5f3-407e-8b41-6836c3b709f1"). InnerVolumeSpecName "kube-api-access-kd4vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:19.865254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:19.865210    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd4vz\" (UniqueName: \"kubernetes.io/projected/3d463a6e-b5f3-407e-8b41-6836c3b709f1-kube-api-access-kd4vz\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:20.330801 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.330768    5879 generic.go:356] "Generic (PLEG): container finished" podID="ebe86781-bd96-4318-8447-767533806fce" containerID="dd6971faa52598af524328c2de5e27bb20dca0fc68e1e0079728196bab797d27" exitCode=60
Nov 02 13:30:20.331507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.330863    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr" event={"ID":"ebe86781-bd96-4318-8447-767533806fce","Type":"ContainerDied","Data":"dd6971faa52598af524328c2de5e27bb20dca0fc68e1e0079728196bab797d27"}
Nov 02 13:30:20.331507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.330947    5879 scope.go:122] "RemoveContainer" containerID="f3b5ead3c96cbb4ef1025a194bcf1c10987c3f02dbe5497634fd7666a8aa7a2e"
Nov 02 13:30:20.331507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.331439    5879 scope.go:122] "RemoveContainer" containerID="dd6971faa52598af524328c2de5e27bb20dca0fc68e1e0079728196bab797d27"
Nov 02 13:30:20.331860 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:20.331822    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr_odh-ai-gateway-infra(ebe86781-bd96-4318-8447-767533806fce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr" podUID="ebe86781-bd96-4318-8447-767533806fce"
Nov 02 13:30:20.341740 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.341710    5879 generic.go:356] "Generic (PLEG): container finished" podID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" containerID="1924969280e85fc5345b5bc8ffbe65dc466bd14976c1d0c144323ac82707556f" exitCode=60
Nov 02 13:30:20.341883 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.341842    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8" event={"ID":"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136","Type":"ContainerDied","Data":"1924969280e85fc5345b5bc8ffbe65dc466bd14976c1d0c144323ac82707556f"}
Nov 02 13:30:20.342315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.342295    5879 scope.go:122] "RemoveContainer" containerID="1924969280e85fc5345b5bc8ffbe65dc466bd14976c1d0c144323ac82707556f"
Nov 02 13:30:20.342676 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:20.342654    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8_odh-ai-gateway-infra(bf789a2a-d8c0-4a66-b2dd-d7a1bf968136)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8" podUID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136"
Nov 02 13:30:20.347918 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.347889    5879 generic.go:356] "Generic (PLEG): container finished" podID="09b683bc-953d-46d1-bc3b-104adddc15f7" containerID="4500171ed6067a11027cbf403d5b09ab6ad41da34a8a8b869b736fb3a2fb4bbf" exitCode=60
Nov 02 13:30:20.348095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.347986    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh" event={"ID":"09b683bc-953d-46d1-bc3b-104adddc15f7","Type":"ContainerDied","Data":"4500171ed6067a11027cbf403d5b09ab6ad41da34a8a8b869b736fb3a2fb4bbf"}
Nov 02 13:30:20.348558 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.348532    5879 scope.go:122] "RemoveContainer" containerID="4500171ed6067a11027cbf403d5b09ab6ad41da34a8a8b869b736fb3a2fb4bbf"
Nov 02 13:30:20.349021 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:20.348948    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh_odh-ai-gateway-infra(09b683bc-953d-46d1-bc3b-104adddc15f7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7"
Nov 02 13:30:20.350831 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.350800    5879 generic.go:356] "Generic (PLEG): container finished" podID="299f06eb-e0df-4927-9d60-738e88a03192" containerID="efd27ae4de97ac42e9f879e1493b485ed5d85daa63d500e98ee49f6249571511" exitCode=60
Nov 02 13:30:20.350928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.350894    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw" event={"ID":"299f06eb-e0df-4927-9d60-738e88a03192","Type":"ContainerDied","Data":"efd27ae4de97ac42e9f879e1493b485ed5d85daa63d500e98ee49f6249571511"}
Nov 02 13:30:20.351421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.351400    5879 scope.go:122] "RemoveContainer" containerID="efd27ae4de97ac42e9f879e1493b485ed5d85daa63d500e98ee49f6249571511"
Nov 02 13:30:20.351870 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:20.351834    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw_odh-ai-gateway-infra(299f06eb-e0df-4927-9d60-738e88a03192)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw" podUID="299f06eb-e0df-4927-9d60-738e88a03192"
Nov 02 13:30:20.355666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.354961    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb" event={"ID":"3d463a6e-b5f3-407e-8b41-6836c3b709f1","Type":"ContainerDied","Data":"f5eb00961c24285d89d42fb3f0f82698a111cd98ddbe621b4397265cd71b017f"}
Nov 02 13:30:20.355666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.355086    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"
Nov 02 13:30:20.364911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.362841    5879 scope.go:122] "RemoveContainer" containerID="14b8be3f989db19eb7a67085f6d955b6d741be02d348fc071965aab0c7acb871"
Nov 02 13:30:20.368897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.368850    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"]
Nov 02 13:30:20.375194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.370776    5879 generic.go:356] "Generic (PLEG): container finished" podID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" containerID="3214eaa966ea7a41b8fea44ad25cb745405e11d53f3a08caea9adc1947178517" exitCode=60
Nov 02 13:30:20.375194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.370834    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k" event={"ID":"4a30bbaa-5aa8-469e-8c1d-1e58a821b310","Type":"ContainerDied","Data":"3214eaa966ea7a41b8fea44ad25cb745405e11d53f3a08caea9adc1947178517"}
Nov 02 13:30:20.375194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.371647    5879 scope.go:122] "RemoveContainer" containerID="3214eaa966ea7a41b8fea44ad25cb745405e11d53f3a08caea9adc1947178517"
Nov 02 13:30:20.384735 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:20.376573    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k_odh-ai-gateway-infra(4a30bbaa-5aa8-469e-8c1d-1e58a821b310)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310"
Nov 02 13:30:20.413217 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.412717    5879 scope.go:122] "RemoveContainer" containerID="e9f099d227f6ab744ebc6c56fc326b3232d6a1cea6dfd48c0244fc1cfb69a77d"
Nov 02 13:30:20.442495 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.442471    5879 scope.go:122] "RemoveContainer" containerID="0f4cd30457088c598df7610e1df202f10696b138d09d2b5e07df4d95db00f3f0"
Nov 02 13:30:20.468685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.468655    5879 scope.go:122] "RemoveContainer" containerID="2353a856c4950f94b8c3afd08ec5424b9aa0b21202c7479c36eafc01d1692ac6"
Nov 02 13:30:20.478901 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.478867    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"]
Nov 02 13:30:20.485020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.484883    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-90cda9-29893770-zcrhb"]
Nov 02 13:30:20.489148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.488386    5879 scope.go:122] "RemoveContainer" containerID="9b186afc8e41adc648bd15fb9c90f2db34fd633b28f9e7c56703a1ec16e2b8c0"
Nov 02 13:30:20.511502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.511468    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"]
Nov 02 13:30:20.596858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.595960    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"
Nov 02 13:30:20.641104 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.641073    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"
Nov 02 13:30:20.645426 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.645401    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"
Nov 02 13:30:20.662340 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.662314    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"
Nov 02 13:30:20.665838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.665809    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"]
Nov 02 13:30:20.680847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.680817    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd1e90e-1165-4bde-8d01-3d37aee5802e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd1e90e-1165-4bde-8d01-3d37aee5802e-maas-api-service-ca\") pod \"acd1e90e-1165-4bde-8d01-3d37aee5802e\" (UID: \"acd1e90e-1165-4bde-8d01-3d37aee5802e\") "
Nov 02 13:30:20.681106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.680887    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-kube-api-access-2rnh4\" (UniqueName: \"kubernetes.io/projected/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-kube-api-access-2rnh4\") pod \"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0\" (UID: \"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0\") "
Nov 02 13:30:20.681106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.681033    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357161b9-27d9-4b14-81de-365061ce42f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357161b9-27d9-4b14-81de-365061ce42f6-maas-api-service-ca\") pod \"357161b9-27d9-4b14-81de-365061ce42f6\" (UID: \"357161b9-27d9-4b14-81de-365061ce42f6\") "
Nov 02 13:30:20.681106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.681080    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-maas-api-service-ca\") pod \"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0\" (UID: \"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0\") "
Nov 02 13:30:20.681304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.681242    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd1e90e-1165-4bde-8d01-3d37aee5802e-kube-api-access-r7bgj\" (UniqueName: \"kubernetes.io/projected/acd1e90e-1165-4bde-8d01-3d37aee5802e-kube-api-access-r7bgj\") pod \"acd1e90e-1165-4bde-8d01-3d37aee5802e\" (UID: \"acd1e90e-1165-4bde-8d01-3d37aee5802e\") "
Nov 02 13:30:20.681457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.681403    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd1e90e-1165-4bde-8d01-3d37aee5802e-maas-api-service-ca" pod "acd1e90e-1165-4bde-8d01-3d37aee5802e" (UID: "acd1e90e-1165-4bde-8d01-3d37aee5802e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:20.681865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.681766    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357161b9-27d9-4b14-81de-365061ce42f6-maas-api-service-ca" pod "357161b9-27d9-4b14-81de-365061ce42f6" (UID: "357161b9-27d9-4b14-81de-365061ce42f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:20.681865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.681827    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-maas-api-service-ca" pod "5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" (UID: "5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:20.682109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.681890    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357161b9-27d9-4b14-81de-365061ce42f6-kube-api-access-nntst\" (UniqueName: \"kubernetes.io/projected/357161b9-27d9-4b14-81de-365061ce42f6-kube-api-access-nntst\") pod \"357161b9-27d9-4b14-81de-365061ce42f6\" (UID: \"357161b9-27d9-4b14-81de-365061ce42f6\") "
Nov 02 13:30:20.682509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.682487    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acd1e90e-1165-4bde-8d01-3d37aee5802e-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:20.682592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.682513    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/357161b9-27d9-4b14-81de-365061ce42f6-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:20.682592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.682527    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:20.684187 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.684162    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357161b9-27d9-4b14-81de-365061ce42f6-kube-api-access-nntst" pod "357161b9-27d9-4b14-81de-365061ce42f6" (UID: "357161b9-27d9-4b14-81de-365061ce42f6"). InnerVolumeSpecName "kube-api-access-nntst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:20.684417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.684379    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd1e90e-1165-4bde-8d01-3d37aee5802e-kube-api-access-r7bgj" pod "acd1e90e-1165-4bde-8d01-3d37aee5802e" (UID: "acd1e90e-1165-4bde-8d01-3d37aee5802e"). InnerVolumeSpecName "kube-api-access-r7bgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:20.684545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.684528    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-kube-api-access-2rnh4" pod "5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" (UID: "5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0"). InnerVolumeSpecName "kube-api-access-2rnh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:20.706781 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.706753    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"]
Nov 02 13:30:20.783538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.783503    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90197636-9fa4-4217-9aaf-7148e5b1479f-kube-api-access-rw925\" (UniqueName: \"kubernetes.io/projected/90197636-9fa4-4217-9aaf-7148e5b1479f-kube-api-access-rw925\") pod \"90197636-9fa4-4217-9aaf-7148e5b1479f\" (UID: \"90197636-9fa4-4217-9aaf-7148e5b1479f\") "
Nov 02 13:30:20.783721 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.783657    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90197636-9fa4-4217-9aaf-7148e5b1479f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90197636-9fa4-4217-9aaf-7148e5b1479f-maas-api-service-ca\") pod \"90197636-9fa4-4217-9aaf-7148e5b1479f\" (UID: \"90197636-9fa4-4217-9aaf-7148e5b1479f\") "
Nov 02 13:30:20.784047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.784019    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90197636-9fa4-4217-9aaf-7148e5b1479f-maas-api-service-ca" pod "90197636-9fa4-4217-9aaf-7148e5b1479f" (UID: "90197636-9fa4-4217-9aaf-7148e5b1479f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:20.784473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.784450    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7bgj\" (UniqueName: \"kubernetes.io/projected/acd1e90e-1165-4bde-8d01-3d37aee5802e-kube-api-access-r7bgj\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:20.784586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.784480    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nntst\" (UniqueName: \"kubernetes.io/projected/357161b9-27d9-4b14-81de-365061ce42f6-kube-api-access-nntst\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:20.784586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.784494    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90197636-9fa4-4217-9aaf-7148e5b1479f-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:20.784586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.784508    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rnh4\" (UniqueName: \"kubernetes.io/projected/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0-kube-api-access-2rnh4\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:20.786085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.786062    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90197636-9fa4-4217-9aaf-7148e5b1479f-kube-api-access-rw925" pod "90197636-9fa4-4217-9aaf-7148e5b1479f" (UID: "90197636-9fa4-4217-9aaf-7148e5b1479f"). InnerVolumeSpecName "kube-api-access-rw925". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:20.885796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:20.885759    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw925\" (UniqueName: \"kubernetes.io/projected/90197636-9fa4-4217-9aaf-7148e5b1479f-kube-api-access-rw925\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.171141 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.171027    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:21.393417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.393383    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"
Nov 02 13:30:21.393417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.393377    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p" event={"ID":"90197636-9fa4-4217-9aaf-7148e5b1479f","Type":"ContainerDied","Data":"8b91937230cb4b5e4736e1c2379c12c1bde6b649794ad58e95e2f6f21d437331"}
Nov 02 13:30:21.394063 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.393500    5879 scope.go:122] "RemoveContainer" containerID="9c784499b10edab0a5bdd4233b2159b8a3a97990b0a77d7d90f77c06b7e03450"
Nov 02 13:30:21.397572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.397544    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7" event={"ID":"5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0","Type":"ContainerDied","Data":"ab923398d0d1fb2da081d2bddfa276fb1a8ef414a139dfb239bc95831da35c62"}
Nov 02 13:30:21.397710 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.397550    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"
Nov 02 13:30:21.399262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.399242    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"
Nov 02 13:30:21.399382 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.399283    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5" event={"ID":"acd1e90e-1165-4bde-8d01-3d37aee5802e","Type":"ContainerDied","Data":"94d272c80931bce94d174e67cc499a77b0b9978ba8fdfa0c4ecb4636be27d5e8"}
Nov 02 13:30:21.405381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.405262    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"
Nov 02 13:30:21.406590 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.406316    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj" event={"ID":"357161b9-27d9-4b14-81de-365061ce42f6","Type":"ContainerDied","Data":"f6e656daa654ef3a5f133a3752ee3d4e43f62bde9f0dfe24a7f64f91663f08cd"}
Nov 02 13:30:21.409670 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.409620    5879 scope.go:122] "RemoveContainer" containerID="d8deb7f74ef3ee20a97638d9840811400e92a457ea7ece8557f285d3545a537d"
Nov 02 13:30:21.434898 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.434314    5879 scope.go:122] "RemoveContainer" containerID="a5ea2b9ee1c0a29664f491327bf6ff0dd96d09776740daf7ad42bbbaaa00a550"
Nov 02 13:30:21.451969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.451584    5879 scope.go:122] "RemoveContainer" containerID="734390a046e2ca2ec2d1e7cddfba2c0a30b6d1e0c6a76a164cdf906b5e46d92d"
Nov 02 13:30:21.501434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.501375    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"]
Nov 02 13:30:21.507344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.506863    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-43be82-29893770-k9h6p"]
Nov 02 13:30:21.541280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.541253    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"]
Nov 02 13:30:21.707505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.707435    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"
Nov 02 13:30:21.710865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.710835    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"
Nov 02 13:30:21.716151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.716121    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"
Nov 02 13:30:21.724575 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.724547    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"
Nov 02 13:30:21.735209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.735182    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"
Nov 02 13:30:21.780106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.780067    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"]
Nov 02 13:30:21.784819 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.784789    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e7561a-29893770-79bg7"]
Nov 02 13:30:21.818430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.818395    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe86781-bd96-4318-8447-767533806fce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe86781-bd96-4318-8447-767533806fce-maas-api-service-ca\") pod \"ebe86781-bd96-4318-8447-767533806fce\" (UID: \"ebe86781-bd96-4318-8447-767533806fce\") "
Nov 02 13:30:21.818613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.818454    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/299f06eb-e0df-4927-9d60-738e88a03192-kube-api-access-v8xbk\" (UniqueName: \"kubernetes.io/projected/299f06eb-e0df-4927-9d60-738e88a03192-kube-api-access-v8xbk\") pod \"299f06eb-e0df-4927-9d60-738e88a03192\" (UID: \"299f06eb-e0df-4927-9d60-738e88a03192\") "
Nov 02 13:30:21.818613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.818499    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-maas-api-service-ca\") pod \"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136\" (UID: \"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136\") "
Nov 02 13:30:21.818613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.818551    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/299f06eb-e0df-4927-9d60-738e88a03192-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/299f06eb-e0df-4927-9d60-738e88a03192-maas-api-service-ca\") pod \"299f06eb-e0df-4927-9d60-738e88a03192\" (UID: \"299f06eb-e0df-4927-9d60-738e88a03192\") "
Nov 02 13:30:21.818782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.818643    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09b683bc-953d-46d1-bc3b-104adddc15f7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09b683bc-953d-46d1-bc3b-104adddc15f7-maas-api-service-ca\") pod \"09b683bc-953d-46d1-bc3b-104adddc15f7\" (UID: \"09b683bc-953d-46d1-bc3b-104adddc15f7\") "
Nov 02 13:30:21.818782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.818683    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-kube-api-access-9qtdf\" (UniqueName: \"kubernetes.io/projected/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-kube-api-access-9qtdf\") pod \"4a30bbaa-5aa8-469e-8c1d-1e58a821b310\" (UID: \"4a30bbaa-5aa8-469e-8c1d-1e58a821b310\") "
Nov 02 13:30:21.818782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.818749    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe86781-bd96-4318-8447-767533806fce-kube-api-access-q7g5h\" (UniqueName: \"kubernetes.io/projected/ebe86781-bd96-4318-8447-767533806fce-kube-api-access-q7g5h\") pod \"ebe86781-bd96-4318-8447-767533806fce\" (UID: \"ebe86781-bd96-4318-8447-767533806fce\") "
Nov 02 13:30:21.818955 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.818822    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09b683bc-953d-46d1-bc3b-104adddc15f7-kube-api-access-226sd\" (UniqueName: \"kubernetes.io/projected/09b683bc-953d-46d1-bc3b-104adddc15f7-kube-api-access-226sd\") pod \"09b683bc-953d-46d1-bc3b-104adddc15f7\" (UID: \"09b683bc-953d-46d1-bc3b-104adddc15f7\") "
Nov 02 13:30:21.818955 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.818868    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-maas-api-service-ca\") pod \"4a30bbaa-5aa8-469e-8c1d-1e58a821b310\" (UID: \"4a30bbaa-5aa8-469e-8c1d-1e58a821b310\") "
Nov 02 13:30:21.818955 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.818867    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe86781-bd96-4318-8447-767533806fce-maas-api-service-ca" pod "ebe86781-bd96-4318-8447-767533806fce" (UID: "ebe86781-bd96-4318-8447-767533806fce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:21.818955 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.818933    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-maas-api-service-ca" pod "bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" (UID: "bf789a2a-d8c0-4a66-b2dd-d7a1bf968136"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:21.818955 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.818936    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299f06eb-e0df-4927-9d60-738e88a03192-maas-api-service-ca" pod "299f06eb-e0df-4927-9d60-738e88a03192" (UID: "299f06eb-e0df-4927-9d60-738e88a03192"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:21.819285 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.819079    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-kube-api-access-dstg6\" (UniqueName: \"kubernetes.io/projected/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-kube-api-access-dstg6\") pod \"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136\" (UID: \"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136\") "
Nov 02 13:30:21.819354 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.819331    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09b683bc-953d-46d1-bc3b-104adddc15f7-maas-api-service-ca" pod "09b683bc-953d-46d1-bc3b-104adddc15f7" (UID: "09b683bc-953d-46d1-bc3b-104adddc15f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:21.819496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.819472    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-maas-api-service-ca" pod "4a30bbaa-5aa8-469e-8c1d-1e58a821b310" (UID: "4a30bbaa-5aa8-469e-8c1d-1e58a821b310"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:21.820022 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.819978    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe86781-bd96-4318-8447-767533806fce-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.820298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.820027    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.820298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.820043    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/299f06eb-e0df-4927-9d60-738e88a03192-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.820298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.820057    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09b683bc-953d-46d1-bc3b-104adddc15f7-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.820298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.820070    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.821218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.821191    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299f06eb-e0df-4927-9d60-738e88a03192-kube-api-access-v8xbk" pod "299f06eb-e0df-4927-9d60-738e88a03192" (UID: "299f06eb-e0df-4927-9d60-738e88a03192"). InnerVolumeSpecName "kube-api-access-v8xbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:21.821395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.821377    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-kube-api-access-9qtdf" pod "4a30bbaa-5aa8-469e-8c1d-1e58a821b310" (UID: "4a30bbaa-5aa8-469e-8c1d-1e58a821b310"). InnerVolumeSpecName "kube-api-access-9qtdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:21.821852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.821834    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe86781-bd96-4318-8447-767533806fce-kube-api-access-q7g5h" pod "ebe86781-bd96-4318-8447-767533806fce" (UID: "ebe86781-bd96-4318-8447-767533806fce"). InnerVolumeSpecName "kube-api-access-q7g5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:21.822290 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.822269    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b683bc-953d-46d1-bc3b-104adddc15f7-kube-api-access-226sd" pod "09b683bc-953d-46d1-bc3b-104adddc15f7" (UID: "09b683bc-953d-46d1-bc3b-104adddc15f7"). InnerVolumeSpecName "kube-api-access-226sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:21.822387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.822348    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-kube-api-access-dstg6" pod "bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" (UID: "bf789a2a-d8c0-4a66-b2dd-d7a1bf968136"). InnerVolumeSpecName "kube-api-access-dstg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:21.822442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.822426    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"]
Nov 02 13:30:21.829265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.829242    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-61e341-29893770-ffgk5"]
Nov 02 13:30:21.867615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.867576    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"]
Nov 02 13:30:21.870781 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.870758    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-25f7a6-29893770-47zkj"]
Nov 02 13:30:21.921694 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.921657    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dstg6\" (UniqueName: \"kubernetes.io/projected/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136-kube-api-access-dstg6\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.921694 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.921697    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8xbk\" (UniqueName: \"kubernetes.io/projected/299f06eb-e0df-4927-9d60-738e88a03192-kube-api-access-v8xbk\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.921904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.921710    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qtdf\" (UniqueName: \"kubernetes.io/projected/4a30bbaa-5aa8-469e-8c1d-1e58a821b310-kube-api-access-9qtdf\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.921904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.921723    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7g5h\" (UniqueName: \"kubernetes.io/projected/ebe86781-bd96-4318-8447-767533806fce-kube-api-access-q7g5h\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:21.921904 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:21.921735    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-226sd\" (UniqueName: \"kubernetes.io/projected/09b683bc-953d-46d1-bc3b-104adddc15f7-kube-api-access-226sd\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:22.018275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.018193    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357161b9-27d9-4b14-81de-365061ce42f6" path="/var/lib/kubelet/pods/357161b9-27d9-4b14-81de-365061ce42f6/volumes"
Nov 02 13:30:22.018777 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.018755    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" path="/var/lib/kubelet/pods/3d463a6e-b5f3-407e-8b41-6836c3b709f1/volumes"
Nov 02 13:30:22.019315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.019295    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" path="/var/lib/kubelet/pods/5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0/volumes"
Nov 02 13:30:22.020079 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.020058    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f" path="/var/lib/kubelet/pods/90197636-9fa4-4217-9aaf-7148e5b1479f/volumes"
Nov 02 13:30:22.020513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.020495    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e" path="/var/lib/kubelet/pods/acd1e90e-1165-4bde-8d01-3d37aee5802e/volumes"
Nov 02 13:30:22.422596 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.422563    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"
Nov 02 13:30:22.423160 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.422597    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k" event={"ID":"4a30bbaa-5aa8-469e-8c1d-1e58a821b310","Type":"ContainerDied","Data":"6fe245358e6e1a17fbc7c08f22bfce672a21927af476092c536aa07ed7c114ee"}
Nov 02 13:30:22.423160 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.422671    5879 scope.go:122] "RemoveContainer" containerID="3214eaa966ea7a41b8fea44ad25cb745405e11d53f3a08caea9adc1947178517"
Nov 02 13:30:22.426781 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.426744    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr" event={"ID":"ebe86781-bd96-4318-8447-767533806fce","Type":"ContainerDied","Data":"f8bdabf86b8d60c429da72dbca7561cb0f3d96508cf0bdda8139d3309a9a5007"}
Nov 02 13:30:22.426911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.426763    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"
Nov 02 13:30:22.428231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.428211    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"
Nov 02 13:30:22.428357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.428240    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8" event={"ID":"bf789a2a-d8c0-4a66-b2dd-d7a1bf968136","Type":"ContainerDied","Data":"459acfb42265480f8f9f020709c49e42e711e52bfd133c2aab3b4816132c848f"}
Nov 02 13:30:22.433838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.433818    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"
Nov 02 13:30:22.433965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.433857    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh" event={"ID":"09b683bc-953d-46d1-bc3b-104adddc15f7","Type":"ContainerDied","Data":"ccdc6096e0de556f0deb59af45066431271886b1955b77b2446208ac718fda83"}
Nov 02 13:30:22.435233 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.435214    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw" event={"ID":"299f06eb-e0df-4927-9d60-738e88a03192","Type":"ContainerDied","Data":"5b8dd1ab8ac6b152edd8beb68ddb19d5592234317bc6977000c5022654276cfa"}
Nov 02 13:30:22.435344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.435237    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"
Nov 02 13:30:22.436398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.436381    5879 scope.go:122] "RemoveContainer" containerID="dd6971faa52598af524328c2de5e27bb20dca0fc68e1e0079728196bab797d27"
Nov 02 13:30:22.446248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.446219    5879 scope.go:122] "RemoveContainer" containerID="1924969280e85fc5345b5bc8ffbe65dc466bd14976c1d0c144323ac82707556f"
Nov 02 13:30:22.456794 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.456775    5879 scope.go:122] "RemoveContainer" containerID="4500171ed6067a11027cbf403d5b09ab6ad41da34a8a8b869b736fb3a2fb4bbf"
Nov 02 13:30:22.469333 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.469300    5879 scope.go:122] "RemoveContainer" containerID="efd27ae4de97ac42e9f879e1493b485ed5d85daa63d500e98ee49f6249571511"
Nov 02 13:30:22.482060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.482036    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"]
Nov 02 13:30:22.485890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.485866    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-ed56b9-29893770-vh7wr"]
Nov 02 13:30:22.524879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.524849    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"]
Nov 02 13:30:22.529828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.529799    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3396f-29893770-dbvkw"]
Nov 02 13:30:22.575524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.575492    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"]
Nov 02 13:30:22.580551 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.580523    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5514e5-29893770-g66l8"]
Nov 02 13:30:22.618873 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.618838    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"]
Nov 02 13:30:22.624496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.624458    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-6416f0-29893770-98t5k"]
Nov 02 13:30:22.662052 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.662020    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"]
Nov 02 13:30:22.668020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.667972    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-95ee26-29893770-296jh"]
Nov 02 13:30:22.905527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:22.905489    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:24.020159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:24.020122    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7" path="/var/lib/kubelet/pods/09b683bc-953d-46d1-bc3b-104adddc15f7/volumes"
Nov 02 13:30:24.020776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:24.020753    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="299f06eb-e0df-4927-9d60-738e88a03192" path="/var/lib/kubelet/pods/299f06eb-e0df-4927-9d60-738e88a03192/volumes"
Nov 02 13:30:24.021247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:24.021226    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" path="/var/lib/kubelet/pods/4a30bbaa-5aa8-469e-8c1d-1e58a821b310/volumes"
Nov 02 13:30:24.022036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:24.022016    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" path="/var/lib/kubelet/pods/bf789a2a-d8c0-4a66-b2dd-d7a1bf968136/volumes"
Nov 02 13:30:24.022480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:24.022463    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe86781-bd96-4318-8447-767533806fce" path="/var/lib/kubelet/pods/ebe86781-bd96-4318-8447-767533806fce/volumes"
Nov 02 13:30:24.335461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:24.335367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:24.557318 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:24.557272    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:26.105957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:26.105921    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:27.883855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:27.883825    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:28.484253 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:28.484217    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:28.621269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:28.621200    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:28.750784 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:28.750697    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:29.121338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:29.121297    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:29.240892 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:29.240859    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:29.453789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:29.453688    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:29.607047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:29.606715    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:29.950970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:29.950937    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:31.020080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:31.020051    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:31.324452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:31.324362    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:31.705262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:31.705233    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:31.843562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:31.843512    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:32.251197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:32.251162    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:32.372841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:32.372807    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:32.998636 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:32.998605    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:33.370806 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:33.370776    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:33.739907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:33.739822    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:34.134887 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:34.134849    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:35.441413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:35.441376    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:37.038765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:37.038724    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:37.254542 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:37.254505    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:37.498287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:37.498257    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:38.837825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:38.837797    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:39.867457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:39.867423    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"]
Nov 02 13:30:40.059518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:40.059487    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:40.198301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:40.198213    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:40.392619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:40.392586    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:41.021472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:41.021427    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:41.378840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:41.378801    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:42.935643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:42.935614    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:43.060484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:43.060448    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:44.562682 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:44.562507    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:44.823308 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:30:44.823218    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:30:44.868776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:44.868718    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp" podUID="a1e903ab-013b-42e2-9a21-3f94d1f7e79e" containerName="payload-processing" containerID="cri-o://504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9" gracePeriod=25
Nov 02 13:30:46.132055 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:46.132021    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:46.759513 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:46.759474    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp" podUID="a1e903ab-013b-42e2-9a21-3f94d1f7e79e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:30:47.821151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:47.821112    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:49.274528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:49.274491    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:49.402489 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:49.402454    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:50.034108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:50.034077    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:50.400986 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:50.400953    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:50.837628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:50.837543    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:51.041976 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.041938    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"]
Nov 02 13:30:51.055594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.055565    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af934d9e-ww4nl"]
Nov 02 13:30:51.257774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.257724    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:51.267286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.267247    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"]
Nov 02 13:30:51.268352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268332    5879 cpu_manager.go:414] "containerMap: removing container" podUID="357161b9-27d9-4b14-81de-365061ce42f6" containerName="cleanup"
Nov 02 13:30:51.268422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268355    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357161b9-27d9-4b14-81de-365061ce42f6" containerName="cleanup"
Nov 02 13:30:51.268422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268368    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5e755e54-3856-4cab-818e-36e2aca4076d" containerName="cleanup"
Nov 02 13:30:51.268422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268376    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e755e54-3856-4cab-818e-36e2aca4076d" containerName="cleanup"
Nov 02 13:30:51.268422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268392    5879 cpu_manager.go:414] "containerMap: removing container" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7" containerName="cleanup"
Nov 02 13:30:51.268422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268400    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7" containerName="cleanup"
Nov 02 13:30:51.268422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268409    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5e755e54-3856-4cab-818e-36e2aca4076d" containerName="cleanup"
Nov 02 13:30:51.268422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268417    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e755e54-3856-4cab-818e-36e2aca4076d" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268431    5879 cpu_manager.go:414] "containerMap: removing container" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268439    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268452    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268459    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268468    5879 cpu_manager.go:414] "containerMap: removing container" podUID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268475    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268484    5879 cpu_manager.go:414] "containerMap: removing container" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268492    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268509    5879 cpu_manager.go:414] "containerMap: removing container" podUID="74a07f2c-977a-4740-b9fb-56ee22098f68" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268517    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a07f2c-977a-4740-b9fb-56ee22098f68" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268528    5879 cpu_manager.go:414] "containerMap: removing container" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268535    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268548    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ebe86781-bd96-4318-8447-767533806fce" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268555    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe86781-bd96-4318-8447-767533806fce" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268564    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ebe86781-bd96-4318-8447-767533806fce" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268571    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe86781-bd96-4318-8447-767533806fce" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268583    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268590    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268613    5879 cpu_manager.go:414] "containerMap: removing container" podUID="357161b9-27d9-4b14-81de-365061ce42f6" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268620    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357161b9-27d9-4b14-81de-365061ce42f6" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268634    5879 cpu_manager.go:414] "containerMap: removing container" podUID="357161b9-27d9-4b14-81de-365061ce42f6" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268641    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357161b9-27d9-4b14-81de-365061ce42f6" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268654    5879 cpu_manager.go:414] "containerMap: removing container" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f" containerName="cleanup"
Nov 02 13:30:51.268663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268660    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268682    5879 cpu_manager.go:414] "containerMap: removing container" podUID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268689    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268704    5879 cpu_manager.go:414] "containerMap: removing container" podUID="299f06eb-e0df-4927-9d60-738e88a03192" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268710    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299f06eb-e0df-4927-9d60-738e88a03192" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268724    5879 cpu_manager.go:414] "containerMap: removing container" podUID="299f06eb-e0df-4927-9d60-738e88a03192" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268730    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299f06eb-e0df-4927-9d60-738e88a03192" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268741    5879 cpu_manager.go:414] "containerMap: removing container" podUID="299f06eb-e0df-4927-9d60-738e88a03192" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268747    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="299f06eb-e0df-4927-9d60-738e88a03192" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268757    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268762    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268773    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268778    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268788    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268793    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268812    5879 cpu_manager.go:414] "containerMap: removing container" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268818    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268828    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268834    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268843    5879 cpu_manager.go:414] "containerMap: removing container" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268850    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268863    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268869    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268883    5879 cpu_manager.go:414] "containerMap: removing container" podUID="048eb183-2cd4-49fc-b6fc-2c4662a29637" containerName="revoke-keys"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268889    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048eb183-2cd4-49fc-b6fc-2c4662a29637" containerName="revoke-keys"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268896    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.268902    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269088    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269100    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="299f06eb-e0df-4927-9d60-738e88a03192" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269116    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269123    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="299f06eb-e0df-4927-9d60-738e88a03192" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269134    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269141    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="299f06eb-e0df-4927-9d60-738e88a03192" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269151    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269159    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269169    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec6b8b5-a0ac-40aa-a266-5d1e16f62ff0" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269176    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269187    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269197    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a07f2c-977a-4740-b9fb-56ee22098f68" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269206    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7edc2c-4ae5-4714-a9b6-1b07c6700083" containerName="main"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269215    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269223    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269230    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe86781-bd96-4318-8447-767533806fce" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269238    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ad3fad-c135-4470-9d9d-429d0d2df2b4" containerName="main"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269248    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="048eb183-2cd4-49fc-b6fc-2c4662a29637" containerName="revoke-keys"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269255    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269264    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="357161b9-27d9-4b14-81de-365061ce42f6" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269272    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e755e54-3856-4cab-818e-36e2aca4076d" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269279    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269287    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e755e54-3856-4cab-818e-36e2aca4076d" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269297    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269307    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269317    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a07f2c-977a-4740-b9fb-56ee22098f68" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269328    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269345    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe86781-bd96-4318-8447-767533806fce" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269355    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe86781-bd96-4318-8447-767533806fce" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269366    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269378    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="357161b9-27d9-4b14-81de-365061ce42f6" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269388    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="357161b9-27d9-4b14-81de-365061ce42f6" containerName="cleanup"
Nov 02 13:30:51.269440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.269398    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" containerName="cleanup"
Nov 02 13:30:51.275100 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.275078    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:51.290188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.290155    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"]
Nov 02 13:30:51.291928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.291883    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:30:51.292313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.292185    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:30:51.336130 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.336090    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04a6b4f0-lh5rh\" (UID: \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:51.336328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.336208    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04a6b4f0-lh5rh\" (UID: \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:51.336328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.336324    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzfpv\" (UniqueName: \"kubernetes.io/projected/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-kube-api-access-mzfpv\") pod \"maas-api-revoke-sub-04a6b4f0-lh5rh\" (UID: \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:51.438044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.437974    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04a6b4f0-lh5rh\" (UID: \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:51.438546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.438174    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzfpv\" (UniqueName: \"kubernetes.io/projected/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-kube-api-access-mzfpv\") pod \"maas-api-revoke-sub-04a6b4f0-lh5rh\" (UID: \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:51.438546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.438389    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04a6b4f0-lh5rh\" (UID: \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:51.439025 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.438972    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04a6b4f0-lh5rh\" (UID: \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:51.440566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.440549    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04a6b4f0-lh5rh\" (UID: \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:51.463492 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.463456    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzfpv\" (UniqueName: \"kubernetes.io/projected/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-kube-api-access-mzfpv\") pod \"maas-api-revoke-sub-04a6b4f0-lh5rh\" (UID: \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:51.589768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.589679    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:51.896554 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:30:51.896520    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341e7f6b_9cb5_4546_b050_9e2bb8f6d130.slice/crio-a2ff1bd206c8c95b12a4d8fcf22bc619f6f3c4f835cb6ca3fee40da7cb789997 WatchSource:0}: Error finding container a2ff1bd206c8c95b12a4d8fcf22bc619f6f3c4f835cb6ca3fee40da7cb789997: Status 404 returned error can't find the container with id a2ff1bd206c8c95b12a4d8fcf22bc619f6f3c4f835cb6ca3fee40da7cb789997
Nov 02 13:30:51.897718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:51.897695    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"]
Nov 02 13:30:52.017747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:52.017711    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="095ba194-3832-46fd-8b37-60ea53723c32" path="/var/lib/kubelet/pods/095ba194-3832-46fd-8b37-60ea53723c32/volumes"
Nov 02 13:30:52.690461 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:52.690411    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh" event={"ID":"341e7f6b-9cb5-4546-b050-9e2bb8f6d130","Type":"ContainerStarted","Data":"66b0bf5c4fbe8906c6aeb91ea2a88c52633fac45e237a7da1f7f8c0d8d4f0e4f"}
Nov 02 13:30:52.690880 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:52.690524    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh" event={"ID":"341e7f6b-9cb5-4546-b050-9e2bb8f6d130","Type":"ContainerStarted","Data":"a2ff1bd206c8c95b12a4d8fcf22bc619f6f3c4f835cb6ca3fee40da7cb789997"}
Nov 02 13:30:52.736722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:52.736661    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh" podStartSLOduration=1.736649172 podStartE2EDuration="1.736649172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:30:52.733302271 +0000 UTC m=+3798.840005121" watchObservedRunningTime="2026-11-02 13:30:52.736649172 +0000 UTC m=+3798.843352022"
Nov 02 13:30:52.854160 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:52.854126    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:53.700147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:53.700111    5879 generic.go:356] "Generic (PLEG): container finished" podID="341e7f6b-9cb5-4546-b050-9e2bb8f6d130" containerID="66b0bf5c4fbe8906c6aeb91ea2a88c52633fac45e237a7da1f7f8c0d8d4f0e4f" exitCode=0
Nov 02 13:30:53.700553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:53.700202    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh" event={"ID":"341e7f6b-9cb5-4546-b050-9e2bb8f6d130","Type":"ContainerDied","Data":"66b0bf5c4fbe8906c6aeb91ea2a88c52633fac45e237a7da1f7f8c0d8d4f0e4f"}
Nov 02 13:30:54.525641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:54.525602    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:54.882191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:54.882163    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:54.987390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:54.987357    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-xks-ca\") pod \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\" (UID: \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\") "
Nov 02 13:30:54.987597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:54.987430    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-kube-api-access-mzfpv\" (UniqueName: \"kubernetes.io/projected/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-kube-api-access-mzfpv\") pod \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\" (UID: \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\") "
Nov 02 13:30:54.987597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:54.987528    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-openshift-ca\") pod \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\" (UID: \"341e7f6b-9cb5-4546-b050-9e2bb8f6d130\") "
Nov 02 13:30:54.987895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:54.987874    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-openshift-ca" pod "341e7f6b-9cb5-4546-b050-9e2bb8f6d130" (UID: "341e7f6b-9cb5-4546-b050-9e2bb8f6d130"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:30:54.988326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:54.988305    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:54.989689 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:54.989668    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-xks-ca" pod "341e7f6b-9cb5-4546-b050-9e2bb8f6d130" (UID: "341e7f6b-9cb5-4546-b050-9e2bb8f6d130"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:30:54.989797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:54.989774    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-kube-api-access-mzfpv" pod "341e7f6b-9cb5-4546-b050-9e2bb8f6d130" (UID: "341e7f6b-9cb5-4546-b050-9e2bb8f6d130"). InnerVolumeSpecName "kube-api-access-mzfpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:30:55.090254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:55.090214    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:55.090254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:55.090248    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzfpv\" (UniqueName: \"kubernetes.io/projected/341e7f6b-9cb5-4546-b050-9e2bb8f6d130-kube-api-access-mzfpv\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:30:55.717163 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:55.717135    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"
Nov 02 13:30:55.717351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:55.717132    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh" event={"ID":"341e7f6b-9cb5-4546-b050-9e2bb8f6d130","Type":"ContainerDied","Data":"a2ff1bd206c8c95b12a4d8fcf22bc619f6f3c4f835cb6ca3fee40da7cb789997"}
Nov 02 13:30:55.717351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:55.717277    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2ff1bd206c8c95b12a4d8fcf22bc619f6f3c4f835cb6ca3fee40da7cb789997"
Nov 02 13:30:55.808035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:55.807984    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"]
Nov 02 13:30:55.923183 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:55.923145    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:56.062683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:56.062590    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:56.658231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:56.658192    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:56.759559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:56.759526    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp" podUID="a1e903ab-013b-42e2-9a21-3f94d1f7e79e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:30:57.796926 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:57.796888    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:30:59.546452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:30:59.546414    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:01.220298 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:01.220267    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:02.033019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.032955    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"]
Nov 02 13:31:02.047009 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.046964    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb994f5e-r2vr4"]
Nov 02 13:31:02.317652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.317575    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:02.449145 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.449114    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:02.456913 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.456882    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"]
Nov 02 13:31:02.457569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457552    5879 cpu_manager.go:414] "containerMap: removing container" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f" containerName="cleanup"
Nov 02 13:31:02.457634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457571    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f" containerName="cleanup"
Nov 02 13:31:02.457634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457591    5879 cpu_manager.go:414] "containerMap: removing container" podUID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" containerName="cleanup"
Nov 02 13:31:02.457634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457596    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" containerName="cleanup"
Nov 02 13:31:02.457634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457604    5879 cpu_manager.go:414] "containerMap: removing container" podUID="341e7f6b-9cb5-4546-b050-9e2bb8f6d130" containerName="revoke-keys"
Nov 02 13:31:02.457634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457609    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341e7f6b-9cb5-4546-b050-9e2bb8f6d130" containerName="revoke-keys"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457640    5879 cpu_manager.go:414] "containerMap: removing container" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457645    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09b683bc-953d-46d1-bc3b-104adddc15f7" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457653    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457657    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457664    5879 cpu_manager.go:414] "containerMap: removing container" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457668    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457678    5879 cpu_manager.go:414] "containerMap: removing container" podUID="74a07f2c-977a-4740-b9fb-56ee22098f68" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457682    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a07f2c-977a-4740-b9fb-56ee22098f68" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457690    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ebe86781-bd96-4318-8447-767533806fce" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457694    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe86781-bd96-4318-8447-767533806fce" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457703    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457707    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d463a6e-b5f3-407e-8b41-6836c3b709f1" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457797    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf789a2a-d8c0-4a66-b2dd-d7a1bf968136" containerName="cleanup"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457806    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="341e7f6b-9cb5-4546-b050-9e2bb8f6d130" containerName="revoke-keys"
Nov 02 13:31:02.457815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457815    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd1e90e-1165-4bde-8d01-3d37aee5802e" containerName="cleanup"
Nov 02 13:31:02.458305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457826    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a30bbaa-5aa8-469e-8c1d-1e58a821b310" containerName="cleanup"
Nov 02 13:31:02.458305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.457832    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="90197636-9fa4-4217-9aaf-7148e5b1479f" containerName="cleanup"
Nov 02 13:31:02.461547 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.461528    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:02.480316 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.480287    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:31:02.480534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.480339    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"]
Nov 02 13:31:02.480968 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.480359    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:31:02.504112 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.504078    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-662453b6-tbsw7\" (UID: \"aecfd315-f66b-41c1-902f-c10db91c7b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:02.504323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.504148    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clckv\" (UniqueName: \"kubernetes.io/projected/aecfd315-f66b-41c1-902f-c10db91c7b96-kube-api-access-clckv\") pod \"maas-api-revoke-sub-662453b6-tbsw7\" (UID: \"aecfd315-f66b-41c1-902f-c10db91c7b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:02.504554 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.504528    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-662453b6-tbsw7\" (UID: \"aecfd315-f66b-41c1-902f-c10db91c7b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:02.605488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.605393    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-662453b6-tbsw7\" (UID: \"aecfd315-f66b-41c1-902f-c10db91c7b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:02.605488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.605444    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clckv\" (UniqueName: \"kubernetes.io/projected/aecfd315-f66b-41c1-902f-c10db91c7b96-kube-api-access-clckv\") pod \"maas-api-revoke-sub-662453b6-tbsw7\" (UID: \"aecfd315-f66b-41c1-902f-c10db91c7b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:02.605760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.605740    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-662453b6-tbsw7\" (UID: \"aecfd315-f66b-41c1-902f-c10db91c7b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:02.606315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.606292    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-662453b6-tbsw7\" (UID: \"aecfd315-f66b-41c1-902f-c10db91c7b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:02.608410 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.608384    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-662453b6-tbsw7\" (UID: \"aecfd315-f66b-41c1-902f-c10db91c7b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:02.631456 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.631426    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clckv\" (UniqueName: \"kubernetes.io/projected/aecfd315-f66b-41c1-902f-c10db91c7b96-kube-api-access-clckv\") pod \"maas-api-revoke-sub-662453b6-tbsw7\" (UID: \"aecfd315-f66b-41c1-902f-c10db91c7b96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:02.776052 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.776017    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:02.950832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:02.950801    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:03.134255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:03.134226    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"]
Nov 02 13:31:03.135125 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:31:03.135097    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaecfd315_f66b_41c1_902f_c10db91c7b96.slice/crio-cbfdbc172eeffe3796c9adde12b446f1e52a0feec96d75cd9b328627ed686a61 WatchSource:0}: Error finding container cbfdbc172eeffe3796c9adde12b446f1e52a0feec96d75cd9b328627ed686a61: Status 404 returned error can't find the container with id cbfdbc172eeffe3796c9adde12b446f1e52a0feec96d75cd9b328627ed686a61
Nov 02 13:31:03.777223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:03.777131    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7" event={"ID":"aecfd315-f66b-41c1-902f-c10db91c7b96","Type":"ContainerStarted","Data":"003fc0080fa4d6f6adc4494165b0259d20b8749c373f680ab91c738303c43620"}
Nov 02 13:31:03.777223 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:03.777189    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7" event={"ID":"aecfd315-f66b-41c1-902f-c10db91c7b96","Type":"ContainerStarted","Data":"cbfdbc172eeffe3796c9adde12b446f1e52a0feec96d75cd9b328627ed686a61"}
Nov 02 13:31:03.826508 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:03.825081    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7" podStartSLOduration=1.825062828 podStartE2EDuration="1.825062828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:31:03.82156266 +0000 UTC m=+3809.928265511" watchObservedRunningTime="2026-11-02 13:31:03.825062828 +0000 UTC m=+3809.931765684"
Nov 02 13:31:04.020571 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:04.020537    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93544e15-db36-48e1-817a-76755f1a355f" path="/var/lib/kubelet/pods/93544e15-db36-48e1-817a-76755f1a355f/volumes"
Nov 02 13:31:04.079191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:04.079104    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:04.600038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:04.598160    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:04.786351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:04.786317    5879 generic.go:356] "Generic (PLEG): container finished" podID="aecfd315-f66b-41c1-902f-c10db91c7b96" containerID="003fc0080fa4d6f6adc4494165b0259d20b8749c373f680ab91c738303c43620" exitCode=0
Nov 02 13:31:04.786792 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:04.786399    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7" event={"ID":"aecfd315-f66b-41c1-902f-c10db91c7b96","Type":"ContainerDied","Data":"003fc0080fa4d6f6adc4494165b0259d20b8749c373f680ab91c738303c43620"}
Nov 02 13:31:05.153270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.153242    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"
Nov 02 13:31:05.239566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.239530    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-plugins-config-volume\") pod \"a1e903ab-013b-42e2-9a21-3f94d1f7e79e\" (UID: \"a1e903ab-013b-42e2-9a21-3f94d1f7e79e\") "
Nov 02 13:31:05.239748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.239735    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-kube-api-access-wtfqq\" (UniqueName: \"kubernetes.io/projected/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-kube-api-access-wtfqq\") pod \"a1e903ab-013b-42e2-9a21-3f94d1f7e79e\" (UID: \"a1e903ab-013b-42e2-9a21-3f94d1f7e79e\") "
Nov 02 13:31:05.239903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.239879    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-plugins-config-volume" pod "a1e903ab-013b-42e2-9a21-3f94d1f7e79e" (UID: "a1e903ab-013b-42e2-9a21-3f94d1f7e79e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:05.240240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.240222    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:05.242246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.242221    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-kube-api-access-wtfqq" pod "a1e903ab-013b-42e2-9a21-3f94d1f7e79e" (UID: "a1e903ab-013b-42e2-9a21-3f94d1f7e79e"). InnerVolumeSpecName "kube-api-access-wtfqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:05.341601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.341564    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtfqq\" (UniqueName: \"kubernetes.io/projected/a1e903ab-013b-42e2-9a21-3f94d1f7e79e-kube-api-access-wtfqq\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:05.796412 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.796379    5879 generic.go:356] "Generic (PLEG): container finished" podID="a1e903ab-013b-42e2-9a21-3f94d1f7e79e" containerID="504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9" exitCode=0
Nov 02 13:31:05.796809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.796438    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"
Nov 02 13:31:05.796809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.796459    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp" event={"ID":"a1e903ab-013b-42e2-9a21-3f94d1f7e79e","Type":"ContainerDied","Data":"504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9"}
Nov 02 13:31:05.796809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.796515    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp" event={"ID":"a1e903ab-013b-42e2-9a21-3f94d1f7e79e","Type":"ContainerDied","Data":"629701a0631b6758656cfe8381eda36a43416ce71b08c312e3e699a307a71231"}
Nov 02 13:31:05.796809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.796539    5879 scope.go:122] "RemoveContainer" containerID="504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9"
Nov 02 13:31:05.809651 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.809619    5879 scope.go:122] "RemoveContainer" containerID="504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9"
Nov 02 13:31:05.809971 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:31:05.809953    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9\": container with ID starting with 504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9 not found: ID does not exist" containerID="504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9"
Nov 02 13:31:05.810067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.809980    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9"} err="failed to get container status \"504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9\": rpc error: code = NotFound desc = could not find container \"504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9\": container with ID starting with 504188bb08e429859301767271ec9adede7a1465ee7ed8b0e142cb5aa68aeaa9 not found: ID does not exist"
Nov 02 13:31:05.915442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.915394    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"]
Nov 02 13:31:05.926717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.926680    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6416f0-7d89b6579f-rlqzp"]
Nov 02 13:31:05.975196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:05.975168    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:06.019350 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.019316    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e903ab-013b-42e2-9a21-3f94d1f7e79e" path="/var/lib/kubelet/pods/a1e903ab-013b-42e2-9a21-3f94d1f7e79e/volumes"
Nov 02 13:31:06.050587 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.050487    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aecfd315-f66b-41c1-902f-c10db91c7b96-kube-api-access-clckv\" (UniqueName: \"kubernetes.io/projected/aecfd315-f66b-41c1-902f-c10db91c7b96-kube-api-access-clckv\") pod \"aecfd315-f66b-41c1-902f-c10db91c7b96\" (UID: \"aecfd315-f66b-41c1-902f-c10db91c7b96\") "
Nov 02 13:31:06.050790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.050619    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-openshift-ca\") pod \"aecfd315-f66b-41c1-902f-c10db91c7b96\" (UID: \"aecfd315-f66b-41c1-902f-c10db91c7b96\") "
Nov 02 13:31:06.050790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.050669    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-xks-ca\") pod \"aecfd315-f66b-41c1-902f-c10db91c7b96\" (UID: \"aecfd315-f66b-41c1-902f-c10db91c7b96\") "
Nov 02 13:31:06.051069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.051038    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-openshift-ca" pod "aecfd315-f66b-41c1-902f-c10db91c7b96" (UID: "aecfd315-f66b-41c1-902f-c10db91c7b96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:06.051389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.051364    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:06.053224 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.053199    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-xks-ca" pod "aecfd315-f66b-41c1-902f-c10db91c7b96" (UID: "aecfd315-f66b-41c1-902f-c10db91c7b96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:31:06.053522 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.053489    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecfd315-f66b-41c1-902f-c10db91c7b96-kube-api-access-clckv" pod "aecfd315-f66b-41c1-902f-c10db91c7b96" (UID: "aecfd315-f66b-41c1-902f-c10db91c7b96"). InnerVolumeSpecName "kube-api-access-clckv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:06.153267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.153219    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clckv\" (UniqueName: \"kubernetes.io/projected/aecfd315-f66b-41c1-902f-c10db91c7b96-kube-api-access-clckv\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:06.153267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.153263    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecfd315-f66b-41c1-902f-c10db91c7b96-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:06.279918 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.279866    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:06.804055 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.804022    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"
Nov 02 13:31:06.804517 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.804024    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7" event={"ID":"aecfd315-f66b-41c1-902f-c10db91c7b96","Type":"ContainerDied","Data":"cbfdbc172eeffe3796c9adde12b446f1e52a0feec96d75cd9b328627ed686a61"}
Nov 02 13:31:06.804517 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:06.804162    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbfdbc172eeffe3796c9adde12b446f1e52a0feec96d75cd9b328627ed686a61"
Nov 02 13:31:07.840272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:07.840237    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:09.458918 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:09.458878    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:10.607049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:10.607013    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:10.833603 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:10.833537    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" containerID="cri-o://aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6" gracePeriod=45
Nov 02 13:31:10.930859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:10.930822    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": dial tcp 10.134.0.49:8000: connect: connection refused"
Nov 02 13:31:11.033972 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.033938    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"]
Nov 02 13:31:11.048482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.048442    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-513d69a5-xrwb6"]
Nov 02 13:31:11.067459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.067421    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:11.125470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.125446    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 13:31:11.224813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.224723    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bee92d4b-5a20-46ef-b035-85b87337be88-kube-api-access-qsqxw\" (UniqueName: \"kubernetes.io/projected/bee92d4b-5a20-46ef-b035-85b87337be88-kube-api-access-qsqxw\") pod \"bee92d4b-5a20-46ef-b035-85b87337be88\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") "
Nov 02 13:31:11.224813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.224773    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-tmp-dir\") pod \"bee92d4b-5a20-46ef-b035-85b87337be88\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") "
Nov 02 13:31:11.225111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.224817    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bee92d4b-5a20-46ef-b035-85b87337be88-tls-certs\" (UniqueName: \"kubernetes.io/secret/bee92d4b-5a20-46ef-b035-85b87337be88-tls-certs\") pod \"bee92d4b-5a20-46ef-b035-85b87337be88\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") "
Nov 02 13:31:11.225111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.224911    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-dshm\") pod \"bee92d4b-5a20-46ef-b035-85b87337be88\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") "
Nov 02 13:31:11.225111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.225030    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-model-cache\") pod \"bee92d4b-5a20-46ef-b035-85b87337be88\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") "
Nov 02 13:31:11.225111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.225064    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-home\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-home\") pod \"bee92d4b-5a20-46ef-b035-85b87337be88\" (UID: \"bee92d4b-5a20-46ef-b035-85b87337be88\") "
Nov 02 13:31:11.226482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.226176    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-tmp-dir" pod "bee92d4b-5a20-46ef-b035-85b87337be88" (UID: "bee92d4b-5a20-46ef-b035-85b87337be88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.227030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.226927    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-home" pod "bee92d4b-5a20-46ef-b035-85b87337be88" (UID: "bee92d4b-5a20-46ef-b035-85b87337be88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.227371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.227338    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-model-cache" pod "bee92d4b-5a20-46ef-b035-85b87337be88" (UID: "bee92d4b-5a20-46ef-b035-85b87337be88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.230657 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.230615    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bee92d4b-5a20-46ef-b035-85b87337be88-tls-certs" pod "bee92d4b-5a20-46ef-b035-85b87337be88" (UID: "bee92d4b-5a20-46ef-b035-85b87337be88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:31:11.230930 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.230901    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bee92d4b-5a20-46ef-b035-85b87337be88-kube-api-access-qsqxw" pod "bee92d4b-5a20-46ef-b035-85b87337be88" (UID: "bee92d4b-5a20-46ef-b035-85b87337be88"). InnerVolumeSpecName "kube-api-access-qsqxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:11.233124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.233088    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-dshm" pod "bee92d4b-5a20-46ef-b035-85b87337be88" (UID: "bee92d4b-5a20-46ef-b035-85b87337be88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:11.327296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.327261    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.327296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.327290    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.327296 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.327299    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.327646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.327308    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsqxw\" (UniqueName: \"kubernetes.io/projected/bee92d4b-5a20-46ef-b035-85b87337be88-kube-api-access-qsqxw\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.327646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.327317    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bee92d4b-5a20-46ef-b035-85b87337be88-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.327646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.327328    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bee92d4b-5a20-46ef-b035-85b87337be88-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:11.843920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.843884    5879 generic.go:356] "Generic (PLEG): container finished" podID="bee92d4b-5a20-46ef-b035-85b87337be88" containerID="aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6" exitCode=0
Nov 02 13:31:11.844472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.843940    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"
Nov 02 13:31:11.844472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.843969    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" event={"ID":"bee92d4b-5a20-46ef-b035-85b87337be88","Type":"ContainerDied","Data":"aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6"}
Nov 02 13:31:11.844472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.844031    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4" event={"ID":"bee92d4b-5a20-46ef-b035-85b87337be88","Type":"ContainerDied","Data":"63686d3f01185411b682d9e09dd780415db9546954524b6685977e08c1272045"}
Nov 02 13:31:11.844472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.844052    5879 scope.go:122] "RemoveContainer" containerID="aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6"
Nov 02 13:31:11.856400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.856379    5879 scope.go:122] "RemoveContainer" containerID="aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6"
Nov 02 13:31:11.856705 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:31:11.856681    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6\": container with ID starting with aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6 not found: ID does not exist" containerID="aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6"
Nov 02 13:31:11.856779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.856714    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6"} err="failed to get container status \"aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6\": rpc error: code = NotFound desc = could not find container \"aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6\": container with ID starting with aec7d31d4ae4bc2daec27ec7517e0bff2c2a383d9cf9a7d5f65b22c888ed33c6 not found: ID does not exist"
Nov 02 13:31:11.905549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.905513    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"]
Nov 02 13:31:11.919573 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:11.919535    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-068756/simulator-w3-068756-kserve-6969d4fc4b-bm6n4"]
Nov 02 13:31:12.018618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:12.018581    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9435ef83-7afb-488c-9a2e-93aef954568b" path="/var/lib/kubelet/pods/9435ef83-7afb-488c-9a2e-93aef954568b/volumes"
Nov 02 13:31:12.019143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:12.019128    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" path="/var/lib/kubelet/pods/bee92d4b-5a20-46ef-b035-85b87337be88/volumes"
Nov 02 13:31:12.597946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:12.597914    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:14.496208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:14.496174    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:14.823335 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:31:14.823232    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:31:16.182556 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:16.182524    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:16.789238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:16.789201    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:17.850150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:17.850113    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:19.454636 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:19.454595    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:21.035072 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:21.035033    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:22.616292 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:22.616263    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:24.233668 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:24.233632    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:25.746964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:25.746926    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:27.536411 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:27.536353    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:27.677805 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:27.677773    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:29.240138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:29.240105    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:30.836384 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:30.836346    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:33.310053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:33.310018    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:33.858613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:33.858574    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:33.993907 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:33.993872    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:34.036383 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:34.036351    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"]
Nov 02 13:31:34.049752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:34.049711    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90c87594-tpcqm"]
Nov 02 13:31:34.437594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:34.437555    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:35.082038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:35.073423    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"]
Nov 02 13:31:35.084272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:35.084237    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c1d5885-9r48g"]
Nov 02 13:31:35.774705 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:35.774672    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:36.033856 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:36.033776    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae" path="/var/lib/kubelet/pods/2cf1b54a-da6e-4e8e-85d5-4ddc4db02eae/volumes"
Nov 02 13:31:36.034786 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:36.034754    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e2e31b0-7ac4-45a3-a927-0670eea8e6df" path="/var/lib/kubelet/pods/5e2e31b0-7ac4-45a3-a927-0670eea8e6df/volumes"
Nov 02 13:31:36.657802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:36.657764    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:38.167392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:38.167358    5879 scope.go:122] "RemoveContainer" containerID="ceefdf6fe9acf7037df57f11a74375e5fd78858689a3ef48e3aec8e667c46c07"
Nov 02 13:31:38.528677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:38.528568    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:39.819069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:39.819033    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:40.528387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:40.528351    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:41.422400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:41.422367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:42.041166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:42.041132    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:42.169659 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:42.169626    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:42.332708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:42.332610    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:43.061430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:43.061385    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:43.203526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:43.203485    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:43.814425 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:43.814372    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:43.958399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:43.958352    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:44.139920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:44.139887    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"]
Nov 02 13:31:44.140723 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:44.140689    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy" containerID="cri-o://74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d" gracePeriod=30
Nov 02 13:31:44.194226 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:44.194189    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:44.827160 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:31:44.827125    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:31:45.631361 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:45.631330    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:47.158300 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:47.158265    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:48.834264 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:48.834232    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:49.429654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.429574    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 13:31:49.598401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.598366    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-certs\") pod \"ea448abc-d63f-463c-9bae-67670be27f32\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") "
Nov 02 13:31:49.598613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.598449    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-data\") pod \"ea448abc-d63f-463c-9bae-67670be27f32\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") "
Nov 02 13:31:49.598613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.598507    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-envoy\") pod \"ea448abc-d63f-463c-9bae-67670be27f32\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") "
Nov 02 13:31:49.598613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.598552    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea448abc-d63f-463c-9bae-67670be27f32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea448abc-d63f-463c-9bae-67670be27f32-istio-podinfo\") pod \"ea448abc-d63f-463c-9bae-67670be27f32\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") "
Nov 02 13:31:49.598613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.598608    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea448abc-d63f-463c-9bae-67670be27f32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea448abc-d63f-463c-9bae-67670be27f32-istiod-ca-cert\") pod \"ea448abc-d63f-463c-9bae-67670be27f32\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") "
Nov 02 13:31:49.598942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.598648    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-kube-api-access-7cxcb\" (UniqueName: \"kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-kube-api-access-7cxcb\") pod \"ea448abc-d63f-463c-9bae-67670be27f32\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") "
Nov 02 13:31:49.598942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.598741    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-certs" pod "ea448abc-d63f-463c-9bae-67670be27f32" (UID: "ea448abc-d63f-463c-9bae-67670be27f32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:49.598942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.598767    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-socket\") pod \"ea448abc-d63f-463c-9bae-67670be27f32\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") "
Nov 02 13:31:49.598942 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.598814    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-credential-socket\") pod \"ea448abc-d63f-463c-9bae-67670be27f32\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") "
Nov 02 13:31:49.599265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.598948    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-data" pod "ea448abc-d63f-463c-9bae-67670be27f32" (UID: "ea448abc-d63f-463c-9bae-67670be27f32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:49.599265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.598962    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-istio-token\" (UniqueName: \"kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-istio-token\") pod \"ea448abc-d63f-463c-9bae-67670be27f32\" (UID: \"ea448abc-d63f-463c-9bae-67670be27f32\") "
Nov 02 13:31:49.599265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.599069    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea448abc-d63f-463c-9bae-67670be27f32-istiod-ca-cert" pod "ea448abc-d63f-463c-9bae-67670be27f32" (UID: "ea448abc-d63f-463c-9bae-67670be27f32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:49.599265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.599166    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-credential-socket" pod "ea448abc-d63f-463c-9bae-67670be27f32" (UID: "ea448abc-d63f-463c-9bae-67670be27f32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:49.599551 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.599281    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-socket" pod "ea448abc-d63f-463c-9bae-67670be27f32" (UID: "ea448abc-d63f-463c-9bae-67670be27f32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:49.600028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.599896    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:49.600028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.599930    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea448abc-d63f-463c-9bae-67670be27f32-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:49.600028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.599946    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:49.600028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.599962    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:49.600028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.599977    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:49.601359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.601330    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-kube-api-access-7cxcb" pod "ea448abc-d63f-463c-9bae-67670be27f32" (UID: "ea448abc-d63f-463c-9bae-67670be27f32"). InnerVolumeSpecName "kube-api-access-7cxcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:49.601486 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.601357    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea448abc-d63f-463c-9bae-67670be27f32-istio-podinfo" pod "ea448abc-d63f-463c-9bae-67670be27f32" (UID: "ea448abc-d63f-463c-9bae-67670be27f32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:31:49.601860 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.601838    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-istio-token" pod "ea448abc-d63f-463c-9bae-67670be27f32" (UID: "ea448abc-d63f-463c-9bae-67670be27f32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:49.602920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.602898    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-envoy" pod "ea448abc-d63f-463c-9bae-67670be27f32" (UID: "ea448abc-d63f-463c-9bae-67670be27f32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:31:49.701240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.701153    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:49.701240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.701180    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea448abc-d63f-463c-9bae-67670be27f32-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:49.701240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.701191    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea448abc-d63f-463c-9bae-67670be27f32-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:49.701240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:49.701199    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cxcb\" (UniqueName: \"kubernetes.io/projected/ea448abc-d63f-463c-9bae-67670be27f32-kube-api-access-7cxcb\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:50.183591 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:50.183556    5879 generic.go:356] "Generic (PLEG): container finished" podID="ea448abc-d63f-463c-9bae-67670be27f32" containerID="74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d" exitCode=0
Nov 02 13:31:50.184146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:50.183660    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" event={"ID":"ea448abc-d63f-463c-9bae-67670be27f32","Type":"ContainerDied","Data":"74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d"}
Nov 02 13:31:50.184146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:50.183746    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb" event={"ID":"ea448abc-d63f-463c-9bae-67670be27f32","Type":"ContainerDied","Data":"9a91292773908949d0da4102ced12255dec9cf889c728a054bf16d1060950f41"}
Nov 02 13:31:50.184146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:50.183782    5879 scope.go:122] "RemoveContainer" containerID="74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d"
Nov 02 13:31:50.184146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:50.183873    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"
Nov 02 13:31:50.199425 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:50.198170    5879 scope.go:122] "RemoveContainer" containerID="74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d"
Nov 02 13:31:50.200444 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:31:50.200409    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d\": container with ID starting with 74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d not found: ID does not exist" containerID="74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d"
Nov 02 13:31:50.200562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:50.200455    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d"} err="failed to get container status \"74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d\": rpc error: code = NotFound desc = could not find container \"74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d\": container with ID starting with 74357ade09e0380a00db8144e199baaf42db31267c02faed9848a4d63b94d67d not found: ID does not exist"
Nov 02 13:31:50.257071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:50.257032    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"]
Nov 02 13:31:50.261197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:50.261168    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-068756-openshift-default-84f69c9cf5-77wtb"]
Nov 02 13:31:50.428624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:50.428592    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:52.019488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:52.019446    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea448abc-d63f-463c-9bae-67670be27f32" path="/var/lib/kubelet/pods/ea448abc-d63f-463c-9bae-67670be27f32/volumes"
Nov 02 13:31:52.093882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:52.093845    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:53.747149 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:53.747112    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:55.311911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.311870    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"]
Nov 02 13:31:55.312817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.312792    5879 cpu_manager.go:414] "containerMap: removing container" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main"
Nov 02 13:31:55.312941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.312820    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main"
Nov 02 13:31:55.312941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.312871    5879 cpu_manager.go:414] "containerMap: removing container" podUID="aecfd315-f66b-41c1-902f-c10db91c7b96" containerName="revoke-keys"
Nov 02 13:31:55.312941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.312880    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecfd315-f66b-41c1-902f-c10db91c7b96" containerName="revoke-keys"
Nov 02 13:31:55.312941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.312910    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy"
Nov 02 13:31:55.312941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.312918    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy"
Nov 02 13:31:55.312941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.312937    5879 cpu_manager.go:414] "containerMap: removing container" podUID="a1e903ab-013b-42e2-9a21-3f94d1f7e79e" containerName="payload-processing"
Nov 02 13:31:55.313277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.312945    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e903ab-013b-42e2-9a21-3f94d1f7e79e" containerName="payload-processing"
Nov 02 13:31:55.313277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.313167    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="bee92d4b-5a20-46ef-b035-85b87337be88" containerName="main"
Nov 02 13:31:55.313277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.313190    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecfd315-f66b-41c1-902f-c10db91c7b96" containerName="revoke-keys"
Nov 02 13:31:55.313277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.313205    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea448abc-d63f-463c-9bae-67670be27f32" containerName="istio-proxy"
Nov 02 13:31:55.313277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.313216    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e903ab-013b-42e2-9a21-3f94d1f7e79e" containerName="payload-processing"
Nov 02 13:31:55.318707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.318681    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:55.337607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.337574    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"]
Nov 02 13:31:55.347282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.347247    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:31:55.347469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.347302    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:31:55.385939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.385905    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c8f7ecc-47r48\" (UID: \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:55.385939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.385951    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dc58\" (UniqueName: \"kubernetes.io/projected/af88dd3b-20a3-4706-8fa7-103c53c5b66e-kube-api-access-8dc58\") pod \"maas-api-revoke-sub-7c8f7ecc-47r48\" (UID: \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:55.386230 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.386207    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c8f7ecc-47r48\" (UID: \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:55.460552 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.460518    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:55.487880 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.487840    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c8f7ecc-47r48\" (UID: \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:55.488168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.487956    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c8f7ecc-47r48\" (UID: \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:55.488168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.488039    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dc58\" (UniqueName: \"kubernetes.io/projected/af88dd3b-20a3-4706-8fa7-103c53c5b66e-kube-api-access-8dc58\") pod \"maas-api-revoke-sub-7c8f7ecc-47r48\" (UID: \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:55.489405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.489373    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c8f7ecc-47r48\" (UID: \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:55.491741 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.491714    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c8f7ecc-47r48\" (UID: \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:55.514895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.514862    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dc58\" (UniqueName: \"kubernetes.io/projected/af88dd3b-20a3-4706-8fa7-103c53c5b66e-kube-api-access-8dc58\") pod \"maas-api-revoke-sub-7c8f7ecc-47r48\" (UID: \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:55.604982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.604897    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"]
Nov 02 13:31:55.609510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.609485    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:31:55.634440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.634386    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"]
Nov 02 13:31:55.634752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.634720    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:55.691320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.691279    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d12afe-wc4rg\" (UID: \"46900508-af5c-462d-9cd5-c5ef358bca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:31:55.691550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.691524    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bsmn\" (UniqueName: \"kubernetes.io/projected/46900508-af5c-462d-9cd5-c5ef358bca6d-kube-api-access-8bsmn\") pod \"maas-api-revoke-sub-68d12afe-wc4rg\" (UID: \"46900508-af5c-462d-9cd5-c5ef358bca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:31:55.691603 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.691575    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d12afe-wc4rg\" (UID: \"46900508-af5c-462d-9cd5-c5ef358bca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:31:55.734544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.734507    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"]
Nov 02 13:31:55.741180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.741158    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:31:55.757563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.757525    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"]
Nov 02 13:31:55.793437 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.793402    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3f703db-srdlf\" (UID: \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:31:55.793621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.793455    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d12afe-wc4rg\" (UID: \"46900508-af5c-462d-9cd5-c5ef358bca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:31:55.793621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.793479    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3f703db-srdlf\" (UID: \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:31:55.793709 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.793664    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bsmn\" (UniqueName: \"kubernetes.io/projected/46900508-af5c-462d-9cd5-c5ef358bca6d-kube-api-access-8bsmn\") pod \"maas-api-revoke-sub-68d12afe-wc4rg\" (UID: \"46900508-af5c-462d-9cd5-c5ef358bca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:31:55.793748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.793710    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d12afe-wc4rg\" (UID: \"46900508-af5c-462d-9cd5-c5ef358bca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:31:55.793748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.793743    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4hr\" (UniqueName: \"kubernetes.io/projected/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-kube-api-access-kn4hr\") pod \"maas-api-revoke-sub-b3f703db-srdlf\" (UID: \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:31:55.794485 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.794460    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d12afe-wc4rg\" (UID: \"46900508-af5c-462d-9cd5-c5ef358bca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:31:55.796061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.796036    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d12afe-wc4rg\" (UID: \"46900508-af5c-462d-9cd5-c5ef358bca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:31:55.819667 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.819625    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bsmn\" (UniqueName: \"kubernetes.io/projected/46900508-af5c-462d-9cd5-c5ef358bca6d-kube-api-access-8bsmn\") pod \"maas-api-revoke-sub-68d12afe-wc4rg\" (UID: \"46900508-af5c-462d-9cd5-c5ef358bca6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:31:55.894927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.894882    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4hr\" (UniqueName: \"kubernetes.io/projected/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-kube-api-access-kn4hr\") pod \"maas-api-revoke-sub-b3f703db-srdlf\" (UID: \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:31:55.895192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.894980    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3f703db-srdlf\" (UID: \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:31:55.895192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.895047    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3f703db-srdlf\" (UID: \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:31:55.895721 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.895696    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3f703db-srdlf\" (UID: \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:31:55.897603 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.897578    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3f703db-srdlf\" (UID: \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:31:55.923081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.923047    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4hr\" (UniqueName: \"kubernetes.io/projected/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-kube-api-access-kn4hr\") pod \"maas-api-revoke-sub-b3f703db-srdlf\" (UID: \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:31:55.925934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.925899    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:31:55.954624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:55.954591    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"]
Nov 02 13:31:55.957625 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:31:55.957584    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf88dd3b_20a3_4706_8fa7_103c53c5b66e.slice/crio-f6d055f1a022141f36d9b7010dd8f4c67bb25143ad936f1627df23b19f7f34f5 WatchSource:0}: Error finding container f6d055f1a022141f36d9b7010dd8f4c67bb25143ad936f1627df23b19f7f34f5: Status 404 returned error can't find the container with id f6d055f1a022141f36d9b7010dd8f4c67bb25143ad936f1627df23b19f7f34f5
Nov 02 13:31:56.061887 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:56.061853    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:31:56.236341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:56.236299    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48" event={"ID":"af88dd3b-20a3-4706-8fa7-103c53c5b66e","Type":"ContainerStarted","Data":"f6d055f1a022141f36d9b7010dd8f4c67bb25143ad936f1627df23b19f7f34f5"}
Nov 02 13:31:56.278845 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:31:56.278810    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46900508_af5c_462d_9cd5_c5ef358bca6d.slice/crio-512230af202c0a4fafd97bef457af2633176e4ef44745f93f4ef507b6091ac05 WatchSource:0}: Error finding container 512230af202c0a4fafd97bef457af2633176e4ef44745f93f4ef507b6091ac05: Status 404 returned error can't find the container with id 512230af202c0a4fafd97bef457af2633176e4ef44745f93f4ef507b6091ac05
Nov 02 13:31:56.279119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:56.279099    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"]
Nov 02 13:31:56.460453 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:56.460423    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"]
Nov 02 13:31:56.484205 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:31:56.484172    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0fc01d_9af0_4ea1_9563_66d78a5e7fe2.slice/crio-21c7e390add6639ef7d03535d331cc11176eb2aaded4ffa2167999ef76fb28a1 WatchSource:0}: Error finding container 21c7e390add6639ef7d03535d331cc11176eb2aaded4ffa2167999ef76fb28a1: Status 404 returned error can't find the container with id 21c7e390add6639ef7d03535d331cc11176eb2aaded4ffa2167999ef76fb28a1
Nov 02 13:31:57.244874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:57.244828    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg" event={"ID":"46900508-af5c-462d-9cd5-c5ef358bca6d","Type":"ContainerStarted","Data":"96fc4cef62ae1d4cf200b99b85d18c4856d5d65dbf84aac7d309ded6c7dfefd9"}
Nov 02 13:31:57.245103 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:57.244929    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg" event={"ID":"46900508-af5c-462d-9cd5-c5ef358bca6d","Type":"ContainerStarted","Data":"512230af202c0a4fafd97bef457af2633176e4ef44745f93f4ef507b6091ac05"}
Nov 02 13:31:57.246599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:57.246571    5879 generic.go:356] "Generic (PLEG): container finished" podID="af88dd3b-20a3-4706-8fa7-103c53c5b66e" containerID="c02dcb828d77c2bea4188723b482e34cc16a42b8ae7786e36b6924adf1e068c7" exitCode=0
Nov 02 13:31:57.246745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:57.246658    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48" event={"ID":"af88dd3b-20a3-4706-8fa7-103c53c5b66e","Type":"ContainerDied","Data":"c02dcb828d77c2bea4188723b482e34cc16a42b8ae7786e36b6924adf1e068c7"}
Nov 02 13:31:57.248336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:57.248312    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf" event={"ID":"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2","Type":"ContainerStarted","Data":"e0c5f0fd55bb55cbcee3f1d6482c4362e887be4734f7f7bee4f3b0ddac18ed5c"}
Nov 02 13:31:57.248440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:57.248350    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf" event={"ID":"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2","Type":"ContainerStarted","Data":"21c7e390add6639ef7d03535d331cc11176eb2aaded4ffa2167999ef76fb28a1"}
Nov 02 13:31:57.253646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:57.253611    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:57.290543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:57.290453    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg" podStartSLOduration=2.290433118 podStartE2EDuration="2.290433118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:31:57.288639414 +0000 UTC m=+3863.395342266" watchObservedRunningTime="2026-11-02 13:31:57.290433118 +0000 UTC m=+3863.397135970"
Nov 02 13:31:57.333863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:57.333811    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf" podStartSLOduration=2.333795994 podStartE2EDuration="2.333795994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:31:57.332043127 +0000 UTC m=+3863.438745979" watchObservedRunningTime="2026-11-02 13:31:57.333795994 +0000 UTC m=+3863.440498844"
Nov 02 13:31:58.257713 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.257680    5879 generic.go:356] "Generic (PLEG): container finished" podID="dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2" containerID="e0c5f0fd55bb55cbcee3f1d6482c4362e887be4734f7f7bee4f3b0ddac18ed5c" exitCode=0
Nov 02 13:31:58.258287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.257771    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf" event={"ID":"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2","Type":"ContainerDied","Data":"e0c5f0fd55bb55cbcee3f1d6482c4362e887be4734f7f7bee4f3b0ddac18ed5c"}
Nov 02 13:31:58.259610 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.259580    5879 generic.go:356] "Generic (PLEG): container finished" podID="46900508-af5c-462d-9cd5-c5ef358bca6d" containerID="96fc4cef62ae1d4cf200b99b85d18c4856d5d65dbf84aac7d309ded6c7dfefd9" exitCode=0
Nov 02 13:31:58.259694 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.259629    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg" event={"ID":"46900508-af5c-462d-9cd5-c5ef358bca6d","Type":"ContainerDied","Data":"96fc4cef62ae1d4cf200b99b85d18c4856d5d65dbf84aac7d309ded6c7dfefd9"}
Nov 02 13:31:58.418035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.418007    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:58.530902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.530814    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af88dd3b-20a3-4706-8fa7-103c53c5b66e-kube-api-access-8dc58\" (UniqueName: \"kubernetes.io/projected/af88dd3b-20a3-4706-8fa7-103c53c5b66e-kube-api-access-8dc58\") pod \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\" (UID: \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\") "
Nov 02 13:31:58.531075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.530946    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-xks-ca\") pod \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\" (UID: \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\") "
Nov 02 13:31:58.531075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.531042    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-openshift-ca\") pod \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\" (UID: \"af88dd3b-20a3-4706-8fa7-103c53c5b66e\") "
Nov 02 13:31:58.531394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.531372    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-openshift-ca" pod "af88dd3b-20a3-4706-8fa7-103c53c5b66e" (UID: "af88dd3b-20a3-4706-8fa7-103c53c5b66e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:58.533276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.533246    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-xks-ca" pod "af88dd3b-20a3-4706-8fa7-103c53c5b66e" (UID: "af88dd3b-20a3-4706-8fa7-103c53c5b66e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:31:58.533436 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.533385    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af88dd3b-20a3-4706-8fa7-103c53c5b66e-kube-api-access-8dc58" pod "af88dd3b-20a3-4706-8fa7-103c53c5b66e" (UID: "af88dd3b-20a3-4706-8fa7-103c53c5b66e"). InnerVolumeSpecName "kube-api-access-8dc58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:58.633484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.633440    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:58.633484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.633483    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af88dd3b-20a3-4706-8fa7-103c53c5b66e-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:58.633712 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.633499    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dc58\" (UniqueName: \"kubernetes.io/projected/af88dd3b-20a3-4706-8fa7-103c53c5b66e-kube-api-access-8dc58\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:58.764562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:58.764523    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:31:59.268178 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.268151    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"
Nov 02 13:31:59.268546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.268193    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48" event={"ID":"af88dd3b-20a3-4706-8fa7-103c53c5b66e","Type":"ContainerDied","Data":"f6d055f1a022141f36d9b7010dd8f4c67bb25143ad936f1627df23b19f7f34f5"}
Nov 02 13:31:59.268546 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.268244    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6d055f1a022141f36d9b7010dd8f4c67bb25143ad936f1627df23b19f7f34f5"
Nov 02 13:31:59.449505 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.449480    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:31:59.480137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.480110    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:31:59.545765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.545676    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-xks-ca\") pod \"46900508-af5c-462d-9cd5-c5ef358bca6d\" (UID: \"46900508-af5c-462d-9cd5-c5ef358bca6d\") "
Nov 02 13:31:59.545765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.545736    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-openshift-ca\") pod \"46900508-af5c-462d-9cd5-c5ef358bca6d\" (UID: \"46900508-af5c-462d-9cd5-c5ef358bca6d\") "
Nov 02 13:31:59.546040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.545842    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-xks-ca\") pod \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\" (UID: \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\") "
Nov 02 13:31:59.546040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.545956    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46900508-af5c-462d-9cd5-c5ef358bca6d-kube-api-access-8bsmn\" (UniqueName: \"kubernetes.io/projected/46900508-af5c-462d-9cd5-c5ef358bca6d-kube-api-access-8bsmn\") pod \"46900508-af5c-462d-9cd5-c5ef358bca6d\" (UID: \"46900508-af5c-462d-9cd5-c5ef358bca6d\") "
Nov 02 13:31:59.546184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.546059    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-openshift-ca\") pod \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\" (UID: \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\") "
Nov 02 13:31:59.546184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.546154    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-openshift-ca" pod "46900508-af5c-462d-9cd5-c5ef358bca6d" (UID: "46900508-af5c-462d-9cd5-c5ef358bca6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:59.546293 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.546258    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-kube-api-access-kn4hr\" (UniqueName: \"kubernetes.io/projected/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-kube-api-access-kn4hr\") pod \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\" (UID: \"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2\") "
Nov 02 13:31:59.547416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.547230    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:59.549222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.549154    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-openshift-ca" pod "dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2" (UID: "dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:31:59.549572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.549546    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-xks-ca" pod "46900508-af5c-462d-9cd5-c5ef358bca6d" (UID: "46900508-af5c-462d-9cd5-c5ef358bca6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:31:59.550606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.550585    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-xks-ca" pod "dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2" (UID: "dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:31:59.550762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.550643    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46900508-af5c-462d-9cd5-c5ef358bca6d-kube-api-access-8bsmn" pod "46900508-af5c-462d-9cd5-c5ef358bca6d" (UID: "46900508-af5c-462d-9cd5-c5ef358bca6d"). InnerVolumeSpecName "kube-api-access-8bsmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:59.551451 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.551428    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-kube-api-access-kn4hr" pod "dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2" (UID: "dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2"). InnerVolumeSpecName "kube-api-access-kn4hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:31:59.648414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.648379    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:59.648414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.648408    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bsmn\" (UniqueName: \"kubernetes.io/projected/46900508-af5c-462d-9cd5-c5ef358bca6d-kube-api-access-8bsmn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:59.648414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.648421    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:59.648669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.648433    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn4hr\" (UniqueName: \"kubernetes.io/projected/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2-kube-api-access-kn4hr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:31:59.648669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:31:59.648443    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46900508-af5c-462d-9cd5-c5ef358bca6d-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:00.277220 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:00.277189    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"
Nov 02 13:32:00.277711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:00.277183    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf" event={"ID":"dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2","Type":"ContainerDied","Data":"21c7e390add6639ef7d03535d331cc11176eb2aaded4ffa2167999ef76fb28a1"}
Nov 02 13:32:00.277711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:00.277351    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21c7e390add6639ef7d03535d331cc11176eb2aaded4ffa2167999ef76fb28a1"
Nov 02 13:32:00.278830 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:00.278802    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"
Nov 02 13:32:00.278951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:00.278832    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg" event={"ID":"46900508-af5c-462d-9cd5-c5ef358bca6d","Type":"ContainerDied","Data":"512230af202c0a4fafd97bef457af2633176e4ef44745f93f4ef507b6091ac05"}
Nov 02 13:32:00.278951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:00.278879    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="512230af202c0a4fafd97bef457af2633176e4ef44745f93f4ef507b6091ac05"
Nov 02 13:32:00.340062 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:00.339964    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:01.954050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:01.953956    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:03.641155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:03.641121    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:05.266645 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.266607    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:05.620254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.620217    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"]
Nov 02 13:32:05.620859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.620846    5879 cpu_manager.go:414] "containerMap: removing container" podUID="46900508-af5c-462d-9cd5-c5ef358bca6d" containerName="revoke-keys"
Nov 02 13:32:05.620909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.620861    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46900508-af5c-462d-9cd5-c5ef358bca6d" containerName="revoke-keys"
Nov 02 13:32:05.620909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.620879    5879 cpu_manager.go:414] "containerMap: removing container" podUID="dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2" containerName="revoke-keys"
Nov 02 13:32:05.620909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.620884    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2" containerName="revoke-keys"
Nov 02 13:32:05.621050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.620912    5879 cpu_manager.go:414] "containerMap: removing container" podUID="af88dd3b-20a3-4706-8fa7-103c53c5b66e" containerName="revoke-keys"
Nov 02 13:32:05.621050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.620917    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af88dd3b-20a3-4706-8fa7-103c53c5b66e" containerName="revoke-keys"
Nov 02 13:32:05.621050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.621028    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="af88dd3b-20a3-4706-8fa7-103c53c5b66e" containerName="revoke-keys"
Nov 02 13:32:05.621050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.621041    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="46900508-af5c-462d-9cd5-c5ef358bca6d" containerName="revoke-keys"
Nov 02 13:32:05.621050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.621049    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2" containerName="revoke-keys"
Nov 02 13:32:05.625001 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.624965    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:05.644248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.644221    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"]
Nov 02 13:32:05.654158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.654131    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:32:05.654341 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.654137    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:32:05.735425 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.735389    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8wrm\" (UniqueName: \"kubernetes.io/projected/a6260358-d500-464b-bf83-b400a5f5d497-kube-api-access-b8wrm\") pod \"maas-api-revoke-sub-a30862bf-vw8v4\" (UID: \"a6260358-d500-464b-bf83-b400a5f5d497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:05.735613 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.735553    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a30862bf-vw8v4\" (UID: \"a6260358-d500-464b-bf83-b400a5f5d497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:05.735752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.735737    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a30862bf-vw8v4\" (UID: \"a6260358-d500-464b-bf83-b400a5f5d497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:05.837417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.837378    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a30862bf-vw8v4\" (UID: \"a6260358-d500-464b-bf83-b400a5f5d497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:05.837601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.837506    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a30862bf-vw8v4\" (UID: \"a6260358-d500-464b-bf83-b400a5f5d497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:05.837601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.837569    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8wrm\" (UniqueName: \"kubernetes.io/projected/a6260358-d500-464b-bf83-b400a5f5d497-kube-api-access-b8wrm\") pod \"maas-api-revoke-sub-a30862bf-vw8v4\" (UID: \"a6260358-d500-464b-bf83-b400a5f5d497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:05.838171 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.838147    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a30862bf-vw8v4\" (UID: \"a6260358-d500-464b-bf83-b400a5f5d497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:05.839921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.839898    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a30862bf-vw8v4\" (UID: \"a6260358-d500-464b-bf83-b400a5f5d497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:05.865015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.864962    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8wrm\" (UniqueName: \"kubernetes.io/projected/a6260358-d500-464b-bf83-b400a5f5d497-kube-api-access-b8wrm\") pod \"maas-api-revoke-sub-a30862bf-vw8v4\" (UID: \"a6260358-d500-464b-bf83-b400a5f5d497\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:05.925690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.925587    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"]
Nov 02 13:32:05.931665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.931632    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:05.939652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.939615    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:05.954514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:05.954481    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"]
Nov 02 13:32:06.040052 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.040019    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9743aa45-ddmg5\" (UID: \"b39179e3-ef04-4cab-b41e-51e0990a2c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:06.040291 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.040269    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfj4s\" (UniqueName: \"kubernetes.io/projected/b39179e3-ef04-4cab-b41e-51e0990a2c84-kube-api-access-mfj4s\") pod \"maas-api-revoke-sub-9743aa45-ddmg5\" (UID: \"b39179e3-ef04-4cab-b41e-51e0990a2c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:06.040448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.040434    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9743aa45-ddmg5\" (UID: \"b39179e3-ef04-4cab-b41e-51e0990a2c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:06.068456 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.068421    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"]
Nov 02 13:32:06.073082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.073062    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:06.093893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.093857    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"]
Nov 02 13:32:06.141584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.141546    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfj4s\" (UniqueName: \"kubernetes.io/projected/b39179e3-ef04-4cab-b41e-51e0990a2c84-kube-api-access-mfj4s\") pod \"maas-api-revoke-sub-9743aa45-ddmg5\" (UID: \"b39179e3-ef04-4cab-b41e-51e0990a2c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:06.141753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.141669    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9743aa45-ddmg5\" (UID: \"b39179e3-ef04-4cab-b41e-51e0990a2c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:06.141819 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.141774    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9743aa45-ddmg5\" (UID: \"b39179e3-ef04-4cab-b41e-51e0990a2c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:06.142390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.142354    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9743aa45-ddmg5\" (UID: \"b39179e3-ef04-4cab-b41e-51e0990a2c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:06.144457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.144437    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9743aa45-ddmg5\" (UID: \"b39179e3-ef04-4cab-b41e-51e0990a2c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:06.167163 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.167128    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfj4s\" (UniqueName: \"kubernetes.io/projected/b39179e3-ef04-4cab-b41e-51e0990a2c84-kube-api-access-mfj4s\") pod \"maas-api-revoke-sub-9743aa45-ddmg5\" (UID: \"b39179e3-ef04-4cab-b41e-51e0990a2c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:06.242612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.242511    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df8d5c7-qvjb2\" (UID: \"b5d29d64-f04e-432f-8136-0315436fad62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:06.242786 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.242673    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9ccm\" (UniqueName: \"kubernetes.io/projected/b5d29d64-f04e-432f-8136-0315436fad62-kube-api-access-j9ccm\") pod \"maas-api-revoke-sub-8df8d5c7-qvjb2\" (UID: \"b5d29d64-f04e-432f-8136-0315436fad62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:06.242870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.242825    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df8d5c7-qvjb2\" (UID: \"b5d29d64-f04e-432f-8136-0315436fad62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:06.251569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.251541    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:06.264264 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.264231    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"]
Nov 02 13:32:06.268248 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:32:06.268218    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6260358_d500_464b_bf83_b400a5f5d497.slice/crio-9fded45b21df6dfba11f5d66727604fc22fc3516e0fc0ffa0d24428b398624e7 WatchSource:0}: Error finding container 9fded45b21df6dfba11f5d66727604fc22fc3516e0fc0ffa0d24428b398624e7: Status 404 returned error can't find the container with id 9fded45b21df6dfba11f5d66727604fc22fc3516e0fc0ffa0d24428b398624e7
Nov 02 13:32:06.322069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.322032    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4" event={"ID":"a6260358-d500-464b-bf83-b400a5f5d497","Type":"ContainerStarted","Data":"9fded45b21df6dfba11f5d66727604fc22fc3516e0fc0ffa0d24428b398624e7"}
Nov 02 13:32:06.344144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.344102    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9ccm\" (UniqueName: \"kubernetes.io/projected/b5d29d64-f04e-432f-8136-0315436fad62-kube-api-access-j9ccm\") pod \"maas-api-revoke-sub-8df8d5c7-qvjb2\" (UID: \"b5d29d64-f04e-432f-8136-0315436fad62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:06.344328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.344269    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df8d5c7-qvjb2\" (UID: \"b5d29d64-f04e-432f-8136-0315436fad62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:06.344416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.344397    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df8d5c7-qvjb2\" (UID: \"b5d29d64-f04e-432f-8136-0315436fad62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:06.345219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.345168    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df8d5c7-qvjb2\" (UID: \"b5d29d64-f04e-432f-8136-0315436fad62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:06.347859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.347838    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df8d5c7-qvjb2\" (UID: \"b5d29d64-f04e-432f-8136-0315436fad62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:06.376679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.376630    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9ccm\" (UniqueName: \"kubernetes.io/projected/b5d29d64-f04e-432f-8136-0315436fad62-kube-api-access-j9ccm\") pod \"maas-api-revoke-sub-8df8d5c7-qvjb2\" (UID: \"b5d29d64-f04e-432f-8136-0315436fad62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:06.390650 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.390608    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:06.573676 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.573630    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"]
Nov 02 13:32:06.784643 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:32:06.784609    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d29d64_f04e_432f_8136_0315436fad62.slice/crio-12a0e5288f89f7954905d1089d25cea58f16b5fe6db5158479c058f9e5e532ea WatchSource:0}: Error finding container 12a0e5288f89f7954905d1089d25cea58f16b5fe6db5158479c058f9e5e532ea: Status 404 returned error can't find the container with id 12a0e5288f89f7954905d1089d25cea58f16b5fe6db5158479c058f9e5e532ea
Nov 02 13:32:06.786265 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.786235    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"]
Nov 02 13:32:06.958704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:06.958669    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:07.331590 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:07.331549    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5" event={"ID":"b39179e3-ef04-4cab-b41e-51e0990a2c84","Type":"ContainerStarted","Data":"6eca7e38263eb037624f990aa916c20d8c636491a625fc57514dae309b46ad08"}
Nov 02 13:32:07.332125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:07.331606    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5" event={"ID":"b39179e3-ef04-4cab-b41e-51e0990a2c84","Type":"ContainerStarted","Data":"fcd8acba4f5f02033c8d70df2e86c90586055958e7ee1a2d7ff8d29504044502"}
Nov 02 13:32:07.333347 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:07.333322    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2" event={"ID":"b5d29d64-f04e-432f-8136-0315436fad62","Type":"ContainerStarted","Data":"cbbb43c08b33388daf166f92fca362eddfd6762e15a7b80f1601e116697df33d"}
Nov 02 13:32:07.333488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:07.333370    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2" event={"ID":"b5d29d64-f04e-432f-8136-0315436fad62","Type":"ContainerStarted","Data":"12a0e5288f89f7954905d1089d25cea58f16b5fe6db5158479c058f9e5e532ea"}
Nov 02 13:32:07.334975 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:07.334954    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4" event={"ID":"a6260358-d500-464b-bf83-b400a5f5d497","Type":"ContainerStarted","Data":"22155dab2a468501318b068c37aa8296dd58dbe2e2f30cd9112be13dd070cbe6"}
Nov 02 13:32:07.378572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:07.378504    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5" podStartSLOduration=2.37848431 podStartE2EDuration="2.37848431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:32:07.374963859 +0000 UTC m=+3873.481666711" watchObservedRunningTime="2026-11-02 13:32:07.37848431 +0000 UTC m=+3873.485187171"
Nov 02 13:32:07.419083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:07.419024    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4" podStartSLOduration=2.419009388 podStartE2EDuration="2.419009388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:32:07.41833073 +0000 UTC m=+3873.525033777" watchObservedRunningTime="2026-11-02 13:32:07.419009388 +0000 UTC m=+3873.525712244"
Nov 02 13:32:07.464447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:07.464389    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2" podStartSLOduration=1.46437476 podStartE2EDuration="1.46437476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:32:07.462542661 +0000 UTC m=+3873.569245512" watchObservedRunningTime="2026-11-02 13:32:07.46437476 +0000 UTC m=+3873.571077610"
Nov 02 13:32:08.344147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:08.344112    5879 generic.go:356] "Generic (PLEG): container finished" podID="b39179e3-ef04-4cab-b41e-51e0990a2c84" containerID="6eca7e38263eb037624f990aa916c20d8c636491a625fc57514dae309b46ad08" exitCode=0
Nov 02 13:32:08.344597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:08.344179    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5" event={"ID":"b39179e3-ef04-4cab-b41e-51e0990a2c84","Type":"ContainerDied","Data":"6eca7e38263eb037624f990aa916c20d8c636491a625fc57514dae309b46ad08"}
Nov 02 13:32:08.346128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:08.346104    5879 generic.go:356] "Generic (PLEG): container finished" podID="b5d29d64-f04e-432f-8136-0315436fad62" containerID="cbbb43c08b33388daf166f92fca362eddfd6762e15a7b80f1601e116697df33d" exitCode=0
Nov 02 13:32:08.346266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:08.346196    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2" event={"ID":"b5d29d64-f04e-432f-8136-0315436fad62","Type":"ContainerDied","Data":"cbbb43c08b33388daf166f92fca362eddfd6762e15a7b80f1601e116697df33d"}
Nov 02 13:32:08.347715 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:08.347694    5879 generic.go:356] "Generic (PLEG): container finished" podID="a6260358-d500-464b-bf83-b400a5f5d497" containerID="22155dab2a468501318b068c37aa8296dd58dbe2e2f30cd9112be13dd070cbe6" exitCode=0
Nov 02 13:32:08.347845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:08.347779    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4" event={"ID":"a6260358-d500-464b-bf83-b400a5f5d497","Type":"ContainerDied","Data":"22155dab2a468501318b068c37aa8296dd58dbe2e2f30cd9112be13dd070cbe6"}
Nov 02 13:32:08.527783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:08.527749    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:09.543302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.543271    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:09.579250 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.579219    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:09.612867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.612797    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:09.692789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.692752    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-xks-ca\") pod \"b5d29d64-f04e-432f-8136-0315436fad62\" (UID: \"b5d29d64-f04e-432f-8136-0315436fad62\") "
Nov 02 13:32:09.692977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.692815    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b39179e3-ef04-4cab-b41e-51e0990a2c84-kube-api-access-mfj4s\" (UniqueName: \"kubernetes.io/projected/b39179e3-ef04-4cab-b41e-51e0990a2c84-kube-api-access-mfj4s\") pod \"b39179e3-ef04-4cab-b41e-51e0990a2c84\" (UID: \"b39179e3-ef04-4cab-b41e-51e0990a2c84\") "
Nov 02 13:32:09.692977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.692949    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-openshift-ca\") pod \"b5d29d64-f04e-432f-8136-0315436fad62\" (UID: \"b5d29d64-f04e-432f-8136-0315436fad62\") "
Nov 02 13:32:09.693135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.693054    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d29d64-f04e-432f-8136-0315436fad62-kube-api-access-j9ccm\" (UniqueName: \"kubernetes.io/projected/b5d29d64-f04e-432f-8136-0315436fad62-kube-api-access-j9ccm\") pod \"b5d29d64-f04e-432f-8136-0315436fad62\" (UID: \"b5d29d64-f04e-432f-8136-0315436fad62\") "
Nov 02 13:32:09.693237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.693217    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-xks-ca\") pod \"b39179e3-ef04-4cab-b41e-51e0990a2c84\" (UID: \"b39179e3-ef04-4cab-b41e-51e0990a2c84\") "
Nov 02 13:32:09.693312 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.693269    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-openshift-ca\") pod \"b39179e3-ef04-4cab-b41e-51e0990a2c84\" (UID: \"b39179e3-ef04-4cab-b41e-51e0990a2c84\") "
Nov 02 13:32:09.693389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.693366    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-openshift-ca" pod "b5d29d64-f04e-432f-8136-0315436fad62" (UID: "b5d29d64-f04e-432f-8136-0315436fad62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:09.693892 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.693863    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:09.694354 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.694258    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-openshift-ca" pod "b39179e3-ef04-4cab-b41e-51e0990a2c84" (UID: "b39179e3-ef04-4cab-b41e-51e0990a2c84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:09.696177 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.696143    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d29d64-f04e-432f-8136-0315436fad62-kube-api-access-j9ccm" pod "b5d29d64-f04e-432f-8136-0315436fad62" (UID: "b5d29d64-f04e-432f-8136-0315436fad62"). InnerVolumeSpecName "kube-api-access-j9ccm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:09.696177 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.696152    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-xks-ca" pod "b5d29d64-f04e-432f-8136-0315436fad62" (UID: "b5d29d64-f04e-432f-8136-0315436fad62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:09.696372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.696335    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b39179e3-ef04-4cab-b41e-51e0990a2c84-kube-api-access-mfj4s" pod "b39179e3-ef04-4cab-b41e-51e0990a2c84" (UID: "b39179e3-ef04-4cab-b41e-51e0990a2c84"). InnerVolumeSpecName "kube-api-access-mfj4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:09.696581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.696555    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-xks-ca" pod "b39179e3-ef04-4cab-b41e-51e0990a2c84" (UID: "b39179e3-ef04-4cab-b41e-51e0990a2c84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:09.794347 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.794311    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6260358-d500-464b-bf83-b400a5f5d497-kube-api-access-b8wrm\" (UniqueName: \"kubernetes.io/projected/a6260358-d500-464b-bf83-b400a5f5d497-kube-api-access-b8wrm\") pod \"a6260358-d500-464b-bf83-b400a5f5d497\" (UID: \"a6260358-d500-464b-bf83-b400a5f5d497\") "
Nov 02 13:32:09.794565 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.794490    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-openshift-ca\") pod \"a6260358-d500-464b-bf83-b400a5f5d497\" (UID: \"a6260358-d500-464b-bf83-b400a5f5d497\") "
Nov 02 13:32:09.794644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.794623    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-xks-ca\") pod \"a6260358-d500-464b-bf83-b400a5f5d497\" (UID: \"a6260358-d500-464b-bf83-b400a5f5d497\") "
Nov 02 13:32:09.794930 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.794901    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-openshift-ca" pod "a6260358-d500-464b-bf83-b400a5f5d497" (UID: "a6260358-d500-464b-bf83-b400a5f5d497"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:09.795119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.795100    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfj4s\" (UniqueName: \"kubernetes.io/projected/b39179e3-ef04-4cab-b41e-51e0990a2c84-kube-api-access-mfj4s\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:09.795175 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.795127    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:09.795175 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.795144    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9ccm\" (UniqueName: \"kubernetes.io/projected/b5d29d64-f04e-432f-8136-0315436fad62-kube-api-access-j9ccm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:09.795175 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.795160    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:09.795316 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.795175    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b39179e3-ef04-4cab-b41e-51e0990a2c84-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:09.795316 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.795189    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d29d64-f04e-432f-8136-0315436fad62-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:09.796631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.796607    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6260358-d500-464b-bf83-b400a5f5d497-kube-api-access-b8wrm" pod "a6260358-d500-464b-bf83-b400a5f5d497" (UID: "a6260358-d500-464b-bf83-b400a5f5d497"). InnerVolumeSpecName "kube-api-access-b8wrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:09.796748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.796710    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-xks-ca" pod "a6260358-d500-464b-bf83-b400a5f5d497" (UID: "a6260358-d500-464b-bf83-b400a5f5d497"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:09.896040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.895961    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6260358-d500-464b-bf83-b400a5f5d497-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:09.896040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:09.896025    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8wrm\" (UniqueName: \"kubernetes.io/projected/a6260358-d500-464b-bf83-b400a5f5d497-kube-api-access-b8wrm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:10.224386 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:10.224301    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:10.365900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:10.365858    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"
Nov 02 13:32:10.366136 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:10.365901    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4" event={"ID":"a6260358-d500-464b-bf83-b400a5f5d497","Type":"ContainerDied","Data":"9fded45b21df6dfba11f5d66727604fc22fc3516e0fc0ffa0d24428b398624e7"}
Nov 02 13:32:10.366136 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:10.365952    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fded45b21df6dfba11f5d66727604fc22fc3516e0fc0ffa0d24428b398624e7"
Nov 02 13:32:10.367742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:10.367720    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"
Nov 02 13:32:10.367902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:10.367772    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5" event={"ID":"b39179e3-ef04-4cab-b41e-51e0990a2c84","Type":"ContainerDied","Data":"fcd8acba4f5f02033c8d70df2e86c90586055958e7ee1a2d7ff8d29504044502"}
Nov 02 13:32:10.367902 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:10.367820    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcd8acba4f5f02033c8d70df2e86c90586055958e7ee1a2d7ff8d29504044502"
Nov 02 13:32:10.369485 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:10.369459    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2" event={"ID":"b5d29d64-f04e-432f-8136-0315436fad62","Type":"ContainerDied","Data":"12a0e5288f89f7954905d1089d25cea58f16b5fe6db5158479c058f9e5e532ea"}
Nov 02 13:32:10.369485 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:10.369481    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"
Nov 02 13:32:10.369664 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:10.369502    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12a0e5288f89f7954905d1089d25cea58f16b5fe6db5158479c058f9e5e532ea"
Nov 02 13:32:12.094611 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:12.094571    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:13.651374 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:13.651287    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:14.822725 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:32:14.822686    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:32:15.256183 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:15.256144    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:16.619937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:16.619871    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"]
Nov 02 13:32:16.750075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:16.750035    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:16.986462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:16.986381    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:17.044395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.044361    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"]
Nov 02 13:32:17.045222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.045202    5879 cpu_manager.go:414] "containerMap: removing container" podUID="b39179e3-ef04-4cab-b41e-51e0990a2c84" containerName="revoke-keys"
Nov 02 13:32:17.045222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.045223    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b39179e3-ef04-4cab-b41e-51e0990a2c84" containerName="revoke-keys"
Nov 02 13:32:17.045358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.045232    5879 cpu_manager.go:414] "containerMap: removing container" podUID="b5d29d64-f04e-432f-8136-0315436fad62" containerName="revoke-keys"
Nov 02 13:32:17.045358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.045238    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d29d64-f04e-432f-8136-0315436fad62" containerName="revoke-keys"
Nov 02 13:32:17.045358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.045249    5879 cpu_manager.go:414] "containerMap: removing container" podUID="a6260358-d500-464b-bf83-b400a5f5d497" containerName="revoke-keys"
Nov 02 13:32:17.045358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.045255    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6260358-d500-464b-bf83-b400a5f5d497" containerName="revoke-keys"
Nov 02 13:32:17.045518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.045409    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="b39179e3-ef04-4cab-b41e-51e0990a2c84" containerName="revoke-keys"
Nov 02 13:32:17.045518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.045418    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d29d64-f04e-432f-8136-0315436fad62" containerName="revoke-keys"
Nov 02 13:32:17.045518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.045425    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6260358-d500-464b-bf83-b400a5f5d497" containerName="revoke-keys"
Nov 02 13:32:17.052319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.052287    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:17.072842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.072814    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"]
Nov 02 13:32:17.093749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.091853    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcrhf\" (UniqueName: \"kubernetes.io/projected/6bb97089-bcb3-4101-ae19-2b932d18344c-kube-api-access-mcrhf\") pod \"maas-api-revoke-sub-7fa8d36f-z5x5k\" (UID: \"6bb97089-bcb3-4101-ae19-2b932d18344c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:17.093749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.092260    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa8d36f-z5x5k\" (UID: \"6bb97089-bcb3-4101-ae19-2b932d18344c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:17.093749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.092313    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa8d36f-z5x5k\" (UID: \"6bb97089-bcb3-4101-ae19-2b932d18344c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:17.103440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.103404    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:32:17.103628 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.103481    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:32:17.130821 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.130785    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"]
Nov 02 13:32:17.193395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.193352    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcrhf\" (UniqueName: \"kubernetes.io/projected/6bb97089-bcb3-4101-ae19-2b932d18344c-kube-api-access-mcrhf\") pod \"maas-api-revoke-sub-7fa8d36f-z5x5k\" (UID: \"6bb97089-bcb3-4101-ae19-2b932d18344c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:17.193600 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.193580    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa8d36f-z5x5k\" (UID: \"6bb97089-bcb3-4101-ae19-2b932d18344c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:17.193644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.193619    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa8d36f-z5x5k\" (UID: \"6bb97089-bcb3-4101-ae19-2b932d18344c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:17.194252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.194227    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa8d36f-z5x5k\" (UID: \"6bb97089-bcb3-4101-ae19-2b932d18344c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:17.196181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.196159    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa8d36f-z5x5k\" (UID: \"6bb97089-bcb3-4101-ae19-2b932d18344c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:17.220167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.220132    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcrhf\" (UniqueName: \"kubernetes.io/projected/6bb97089-bcb3-4101-ae19-2b932d18344c-kube-api-access-mcrhf\") pod \"maas-api-revoke-sub-7fa8d36f-z5x5k\" (UID: \"6bb97089-bcb3-4101-ae19-2b932d18344c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:17.233381 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.233341    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:17.372855 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.372816    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:17.683046 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:32:17.683010    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb97089_bcb3_4101_ae19_2b932d18344c.slice/crio-6c29b1c4177053b40eb448e6443a50bf34ca846f5a33c77280068319aafbe1d3 WatchSource:0}: Error finding container 6c29b1c4177053b40eb448e6443a50bf34ca846f5a33c77280068319aafbe1d3: Status 404 returned error can't find the container with id 6c29b1c4177053b40eb448e6443a50bf34ca846f5a33c77280068319aafbe1d3
Nov 02 13:32:17.683977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:17.683952    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"]
Nov 02 13:32:18.431226 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:18.431183    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k" event={"ID":"6bb97089-bcb3-4101-ae19-2b932d18344c","Type":"ContainerStarted","Data":"b4ed7d9710553938d45744afb1c9122c90fa2d2d7f4df9827906afe7b6b84c73"}
Nov 02 13:32:18.431424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:18.431248    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k" event={"ID":"6bb97089-bcb3-4101-ae19-2b932d18344c","Type":"ContainerStarted","Data":"6c29b1c4177053b40eb448e6443a50bf34ca846f5a33c77280068319aafbe1d3"}
Nov 02 13:32:18.476332 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:18.476273    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k" podStartSLOduration=1.4762569509999999 podStartE2EDuration="1.476256951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:32:18.474087596 +0000 UTC m=+3884.580790448" watchObservedRunningTime="2026-11-02 13:32:18.476256951 +0000 UTC m=+3884.582959834"
Nov 02 13:32:18.818709 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:18.818674    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:19.442459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:19.442419    5879 generic.go:356] "Generic (PLEG): container finished" podID="6bb97089-bcb3-4101-ae19-2b932d18344c" containerID="b4ed7d9710553938d45744afb1c9122c90fa2d2d7f4df9827906afe7b6b84c73" exitCode=0
Nov 02 13:32:19.442654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:19.442473    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k" event={"ID":"6bb97089-bcb3-4101-ae19-2b932d18344c","Type":"ContainerDied","Data":"b4ed7d9710553938d45744afb1c9122c90fa2d2d7f4df9827906afe7b6b84c73"}
Nov 02 13:32:20.443624 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:20.443588    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:20.623950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:20.623916    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:20.754938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:20.754852    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-openshift-ca\") pod \"6bb97089-bcb3-4101-ae19-2b932d18344c\" (UID: \"6bb97089-bcb3-4101-ae19-2b932d18344c\") "
Nov 02 13:32:20.755137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:20.754965    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-xks-ca\") pod \"6bb97089-bcb3-4101-ae19-2b932d18344c\" (UID: \"6bb97089-bcb3-4101-ae19-2b932d18344c\") "
Nov 02 13:32:20.755137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:20.755041    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb97089-bcb3-4101-ae19-2b932d18344c-kube-api-access-mcrhf\" (UniqueName: \"kubernetes.io/projected/6bb97089-bcb3-4101-ae19-2b932d18344c-kube-api-access-mcrhf\") pod \"6bb97089-bcb3-4101-ae19-2b932d18344c\" (UID: \"6bb97089-bcb3-4101-ae19-2b932d18344c\") "
Nov 02 13:32:20.755288 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:20.755261    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-openshift-ca" pod "6bb97089-bcb3-4101-ae19-2b932d18344c" (UID: "6bb97089-bcb3-4101-ae19-2b932d18344c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:20.755559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:20.755535    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:20.757180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:20.757154    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-xks-ca" pod "6bb97089-bcb3-4101-ae19-2b932d18344c" (UID: "6bb97089-bcb3-4101-ae19-2b932d18344c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:20.757270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:20.757221    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb97089-bcb3-4101-ae19-2b932d18344c-kube-api-access-mcrhf" pod "6bb97089-bcb3-4101-ae19-2b932d18344c" (UID: "6bb97089-bcb3-4101-ae19-2b932d18344c"). InnerVolumeSpecName "kube-api-access-mcrhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:20.857469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:20.857429    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb97089-bcb3-4101-ae19-2b932d18344c-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:20.857469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:20.857461    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcrhf\" (UniqueName: \"kubernetes.io/projected/6bb97089-bcb3-4101-ae19-2b932d18344c-kube-api-access-mcrhf\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:21.333143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:21.333107    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"]
Nov 02 13:32:21.401475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:21.401433    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:21.464715 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:21.464686    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"
Nov 02 13:32:21.465214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:21.464735    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k" event={"ID":"6bb97089-bcb3-4101-ae19-2b932d18344c","Type":"ContainerDied","Data":"6c29b1c4177053b40eb448e6443a50bf34ca846f5a33c77280068319aafbe1d3"}
Nov 02 13:32:21.465214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:21.464796    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c29b1c4177053b40eb448e6443a50bf34ca846f5a33c77280068319aafbe1d3"
Nov 02 13:32:21.827308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:21.827256    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:22.147708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:22.147663    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:23.757138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:23.757099    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:25.340572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:25.340533    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:27.003737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.003629    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:27.344670 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.344583    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"]
Nov 02 13:32:27.381695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.381661    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"]
Nov 02 13:32:27.501825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.501789    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:27.526090 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.526054    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"]
Nov 02 13:32:27.527205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.527173    5879 cpu_manager.go:414] "containerMap: removing container" podUID="6bb97089-bcb3-4101-ae19-2b932d18344c" containerName="revoke-keys"
Nov 02 13:32:27.527205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.527199    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb97089-bcb3-4101-ae19-2b932d18344c" containerName="revoke-keys"
Nov 02 13:32:27.527396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.527373    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb97089-bcb3-4101-ae19-2b932d18344c" containerName="revoke-keys"
Nov 02 13:32:27.531929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.531903    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:27.547686 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.547651    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"]
Nov 02 13:32:27.548471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.548443    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:32:27.548608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.548488    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:32:27.618078 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.618045    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:27.653472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.653427    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fspj\" (UniqueName: \"kubernetes.io/projected/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-kube-api-access-6fspj\") pod \"maas-api-revoke-sub-3cb70a73-7wrtq\" (UID: \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:27.653663 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.653566    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cb70a73-7wrtq\" (UID: \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:27.653732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.653718    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cb70a73-7wrtq\" (UID: \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:27.754662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.754621    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fspj\" (UniqueName: \"kubernetes.io/projected/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-kube-api-access-6fspj\") pod \"maas-api-revoke-sub-3cb70a73-7wrtq\" (UID: \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:27.754874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.754700    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cb70a73-7wrtq\" (UID: \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:27.754874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.754806    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cb70a73-7wrtq\" (UID: \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:27.755582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.755559    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cb70a73-7wrtq\" (UID: \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:27.757325 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.757302    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cb70a73-7wrtq\" (UID: \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:27.781342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.781311    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fspj\" (UniqueName: \"kubernetes.io/projected/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-kube-api-access-6fspj\") pod \"maas-api-revoke-sub-3cb70a73-7wrtq\" (UID: \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:27.850248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:27.850203    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:28.179040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:28.178907    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"]
Nov 02 13:32:28.519216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:28.519180    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq" event={"ID":"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a","Type":"ContainerStarted","Data":"5a64f12b1e458f731c4f6e7628303e891168fdfa17ee4545b061dbf54c0159c9"}
Nov 02 13:32:28.685065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:28.685035    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:28.806921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:28.806826    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:29.349928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:29.349887    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"]
Nov 02 13:32:29.437732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:29.437700    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:29.530353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:29.530258    5879 generic.go:356] "Generic (PLEG): container finished" podID="8bebb2ec-9564-4e70-9e56-443a8c8f9a8a" containerID="562add3f1640e7c523c206eeacd4e394fb1ee648d42f9edf8114c8d3b174e9b3" exitCode=0
Nov 02 13:32:29.530537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:29.530405    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq" event={"ID":"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a","Type":"ContainerDied","Data":"562add3f1640e7c523c206eeacd4e394fb1ee648d42f9edf8114c8d3b174e9b3"}
Nov 02 13:32:30.658397 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:30.658355    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:30.724837 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:30.724802    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:30.802578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:30.802490    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-xks-ca\") pod \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\" (UID: \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\") "
Nov 02 13:32:30.802578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:30.802558    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-openshift-ca\") pod \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\" (UID: \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\") "
Nov 02 13:32:30.802815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:30.802610    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-kube-api-access-6fspj\" (UniqueName: \"kubernetes.io/projected/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-kube-api-access-6fspj\") pod \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\" (UID: \"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a\") "
Nov 02 13:32:30.803035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:30.802985    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-openshift-ca" pod "8bebb2ec-9564-4e70-9e56-443a8c8f9a8a" (UID: "8bebb2ec-9564-4e70-9e56-443a8c8f9a8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:30.803460 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:30.803436    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:30.805610 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:30.805579    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-kube-api-access-6fspj" pod "8bebb2ec-9564-4e70-9e56-443a8c8f9a8a" (UID: "8bebb2ec-9564-4e70-9e56-443a8c8f9a8a"). InnerVolumeSpecName "kube-api-access-6fspj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:30.805610 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:30.805589    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-xks-ca" pod "8bebb2ec-9564-4e70-9e56-443a8c8f9a8a" (UID: "8bebb2ec-9564-4e70-9e56-443a8c8f9a8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:30.905306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:30.905254    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:30.905306 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:30.905286    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fspj\" (UniqueName: \"kubernetes.io/projected/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a-kube-api-access-6fspj\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:31.547740 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:31.547713    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"
Nov 02 13:32:31.547906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:31.547777    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq" event={"ID":"8bebb2ec-9564-4e70-9e56-443a8c8f9a8a","Type":"ContainerDied","Data":"5a64f12b1e458f731c4f6e7628303e891168fdfa17ee4545b061dbf54c0159c9"}
Nov 02 13:32:31.547906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:31.547826    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a64f12b1e458f731c4f6e7628303e891168fdfa17ee4545b061dbf54c0159c9"
Nov 02 13:32:31.658464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:31.658421    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8" podUID="fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" containerName="main" containerID="cri-o://3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d" gracePeriod=45
Nov 02 13:32:31.716169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:31.716126    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8" podUID="fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" containerName="main" probeResult="failure" output="Get \"https://10.134.1.103:8000/ready\": dial tcp 10.134.1.103:8000: connect: connection refused"
Nov 02 13:32:31.956587 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:31.956547    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:32:32.023552 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.023514    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-dshm\") pod \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") "
Nov 02 13:32:32.023766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.023596    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-kube-api-access-67hhd\" (UniqueName: \"kubernetes.io/projected/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-kube-api-access-67hhd\") pod \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") "
Nov 02 13:32:32.023766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.023628    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-home\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-home\") pod \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") "
Nov 02 13:32:32.023924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.023804    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tmp-dir\") pod \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") "
Nov 02 13:32:32.023984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.023952    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-model-cache\") pod \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") "
Nov 02 13:32:32.024122 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.024039    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tls-certs\") pod \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\" (UID: \"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb\") "
Nov 02 13:32:32.024122 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.024041    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-home" pod "fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" (UID: "fe7c00e6-5bde-4c59-bda7-6ea5894cabfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:32.024122 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.024087    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tmp-dir" pod "fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" (UID: "fe7c00e6-5bde-4c59-bda7-6ea5894cabfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:32.024295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.024222    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-model-cache" pod "fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" (UID: "fe7c00e6-5bde-4c59-bda7-6ea5894cabfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:32.024597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.024575    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:32.024597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.024596    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:32.024800 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.024605    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:32.026982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.026952    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tls-certs" pod "fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" (UID: "fe7c00e6-5bde-4c59-bda7-6ea5894cabfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:32.027345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.027323    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-kube-api-access-67hhd" pod "fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" (UID: "fe7c00e6-5bde-4c59-bda7-6ea5894cabfb"). InnerVolumeSpecName "kube-api-access-67hhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:32.027625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.027604    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-dshm" pod "fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" (UID: "fe7c00e6-5bde-4c59-bda7-6ea5894cabfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:32.125740 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.125703    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67hhd\" (UniqueName: \"kubernetes.io/projected/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-kube-api-access-67hhd\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:32.125740 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.125738    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:32.125740 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.125748    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:32.154948 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.154904    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2" podUID="b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed" containerName="main" containerID="cri-o://7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61" gracePeriod=45
Nov 02 13:32:32.345289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.345238    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8" podUID="f44570e6-9d69-418d-8a2a-8dcbe46857f2" containerName="payload-processing" containerID="cri-o://a374ab583818a172e51deab2cc508570f1062a60e0236ec7ef6bf10a5d9a16a5" gracePeriod=25
Nov 02 13:32:32.382034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.381922    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn" podUID="8bee752b-e2a7-4348-b924-8615b18822d7" containerName="payload-pre-processing" containerID="cri-o://3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796" gracePeriod=25
Nov 02 13:32:32.382275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.382249    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:32.433059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.432976    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:32:32.530050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.530018    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tmp-dir\") pod \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") "
Nov 02 13:32:32.530257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.530086    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-dshm\") pod \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") "
Nov 02 13:32:32.530257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.530216    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tls-certs\") pod \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") "
Nov 02 13:32:32.530257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.530252    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-model-cache\") pod \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") "
Nov 02 13:32:32.530438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.530309    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-kube-api-access-22fk9\" (UniqueName: \"kubernetes.io/projected/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-kube-api-access-22fk9\") pod \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") "
Nov 02 13:32:32.530438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.530346    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-home\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-home\") pod \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\" (UID: \"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed\") "
Nov 02 13:32:32.530438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.530345    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tmp-dir" pod "b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed" (UID: "b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:32.530644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.530596    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-model-cache" pod "b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed" (UID: "b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:32.530704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.530677    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-home" pod "b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed" (UID: "b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:32.530899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.530846    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:32.530899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.530864    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:32.530899 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.530872    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:32.532585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.532549    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tls-certs" pod "b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed" (UID: "b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:32.532709 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.532641    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-dshm" pod "b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed" (UID: "b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:32.532709 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.532666    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-kube-api-access-22fk9" pod "b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed" (UID: "b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed"). InnerVolumeSpecName "kube-api-access-22fk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:32.556395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.556357    5879 generic.go:356] "Generic (PLEG): container finished" podID="b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed" containerID="7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61" exitCode=0
Nov 02 13:32:32.556577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.556455    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"
Nov 02 13:32:32.556577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.556495    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2" event={"ID":"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed","Type":"ContainerDied","Data":"7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61"}
Nov 02 13:32:32.556577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.556542    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2" event={"ID":"b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed","Type":"ContainerDied","Data":"cc47927f909c7d742b4318f3c7ce54e3273e979d7f9844724df08b385069ab34"}
Nov 02 13:32:32.556577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.556567    5879 scope.go:122] "RemoveContainer" containerID="7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61"
Nov 02 13:32:32.558221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.558170    5879 generic.go:356] "Generic (PLEG): container finished" podID="fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" containerID="3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d" exitCode=0
Nov 02 13:32:32.558221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.558217    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"
Nov 02 13:32:32.558400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.558282    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8" event={"ID":"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb","Type":"ContainerDied","Data":"3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d"}
Nov 02 13:32:32.558453 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.558421    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8" event={"ID":"fe7c00e6-5bde-4c59-bda7-6ea5894cabfb","Type":"ContainerDied","Data":"fa76baf2ab1ebdd556e0246a028deb57253eefa72295a1417b00a1043cd12fae"}
Nov 02 13:32:32.568102 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.568082    5879 scope.go:122] "RemoveContainer" containerID="7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61"
Nov 02 13:32:32.568453 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:32:32.568432    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61\": container with ID starting with 7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61 not found: ID does not exist" containerID="7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61"
Nov 02 13:32:32.568534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.568465    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61"} err="failed to get container status \"7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61\": rpc error: code = NotFound desc = could not find container \"7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61\": container with ID starting with 7b710d1c84159b38368db470d994d15d9f67f4c70ab068e4bc41136641defa61 not found: ID does not exist"
Nov 02 13:32:32.568534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.568490    5879 scope.go:122] "RemoveContainer" containerID="3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d"
Nov 02 13:32:32.579059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.579039    5879 scope.go:122] "RemoveContainer" containerID="3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d"
Nov 02 13:32:32.579479 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:32:32.579455    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d\": container with ID starting with 3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d not found: ID does not exist" containerID="3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d"
Nov 02 13:32:32.579563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.579488    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d"} err="failed to get container status \"3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d\": rpc error: code = NotFound desc = could not find container \"3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d\": container with ID starting with 3a228e1dd58638f23ae1148a951aab8df306ce5e36feb822f5758cbbef25393d not found: ID does not exist"
Nov 02 13:32:32.614654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.614617    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"]
Nov 02 13:32:32.618409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.618379    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-8032f3/rate-test-model-8032f3-kserve-758c77fd5c-xrlr8"]
Nov 02 13:32:32.631896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.631864    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:32.631896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.631890    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:32.631896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.631902    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22fk9\" (UniqueName: \"kubernetes.io/projected/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed-kube-api-access-22fk9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:32.655580 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.655472    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"]
Nov 02 13:32:32.658920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.658897    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a6f2c0/rate-test-model-a6f2c0-kserve-bc4588bd9-j8fd2"]
Nov 02 13:32:32.684073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.684032    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn" podUID="8bee752b-e2a7-4348-b924-8615b18822d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:32:32.684259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:32.684036    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8" podUID="f44570e6-9d69-418d-8a2a-8dcbe46857f2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:32:34.018626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:34.018591    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed" path="/var/lib/kubelet/pods/b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed/volumes"
Nov 02 13:32:34.019256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:34.019235    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" path="/var/lib/kubelet/pods/fe7c00e6-5bde-4c59-bda7-6ea5894cabfb/volumes"
Nov 02 13:32:34.168268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:34.168239    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:34.952788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:34.952740    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:35.848165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:35.848125    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:36.358209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.358174    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt" podUID="a5ae966c-6997-446c-a6b6-52393972b61c" containerName="main" containerID="cri-o://61df33f8c78f45658bb0f858fc3360b04eb74f6d53f6411b9487c4b679725daa" gracePeriod=45
Nov 02 13:32:36.595529 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.595490    5879 generic.go:356] "Generic (PLEG): container finished" podID="a5ae966c-6997-446c-a6b6-52393972b61c" containerID="61df33f8c78f45658bb0f858fc3360b04eb74f6d53f6411b9487c4b679725daa" exitCode=0
Nov 02 13:32:36.595708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.595558    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt" event={"ID":"a5ae966c-6997-446c-a6b6-52393972b61c","Type":"ContainerDied","Data":"61df33f8c78f45658bb0f858fc3360b04eb74f6d53f6411b9487c4b679725daa"}
Nov 02 13:32:36.661817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.661793    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:32:36.789443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.789403    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-model-cache\") pod \"a5ae966c-6997-446c-a6b6-52393972b61c\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") "
Nov 02 13:32:36.789673 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.789491    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-home\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-home\") pod \"a5ae966c-6997-446c-a6b6-52393972b61c\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") "
Nov 02 13:32:36.789673 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.789530    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-dshm\") pod \"a5ae966c-6997-446c-a6b6-52393972b61c\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") "
Nov 02 13:32:36.789810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.789733    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-tmp-dir\") pod \"a5ae966c-6997-446c-a6b6-52393972b61c\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") "
Nov 02 13:32:36.789810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.789769    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-home" pod "a5ae966c-6997-446c-a6b6-52393972b61c" (UID: "a5ae966c-6997-446c-a6b6-52393972b61c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:36.789810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.789781    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-model-cache" pod "a5ae966c-6997-446c-a6b6-52393972b61c" (UID: "a5ae966c-6997-446c-a6b6-52393972b61c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:36.789810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.789799    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ae966c-6997-446c-a6b6-52393972b61c-kube-api-access-sffc6\" (UniqueName: \"kubernetes.io/projected/a5ae966c-6997-446c-a6b6-52393972b61c-kube-api-access-sffc6\") pod \"a5ae966c-6997-446c-a6b6-52393972b61c\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") "
Nov 02 13:32:36.790060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.789948    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5ae966c-6997-446c-a6b6-52393972b61c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ae966c-6997-446c-a6b6-52393972b61c-tls-certs\") pod \"a5ae966c-6997-446c-a6b6-52393972b61c\" (UID: \"a5ae966c-6997-446c-a6b6-52393972b61c\") "
Nov 02 13:32:36.790060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.790040    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-tmp-dir" pod "a5ae966c-6997-446c-a6b6-52393972b61c" (UID: "a5ae966c-6997-446c-a6b6-52393972b61c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:36.790536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.790514    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:36.790652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.790538    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:36.790652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.790550    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:36.792271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.792245    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ae966c-6997-446c-a6b6-52393972b61c-tls-certs" pod "a5ae966c-6997-446c-a6b6-52393972b61c" (UID: "a5ae966c-6997-446c-a6b6-52393972b61c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:36.792399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.792377    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-dshm" pod "a5ae966c-6997-446c-a6b6-52393972b61c" (UID: "a5ae966c-6997-446c-a6b6-52393972b61c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:36.792464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.792422    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ae966c-6997-446c-a6b6-52393972b61c-kube-api-access-sffc6" pod "a5ae966c-6997-446c-a6b6-52393972b61c" (UID: "a5ae966c-6997-446c-a6b6-52393972b61c"). InnerVolumeSpecName "kube-api-access-sffc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:36.891441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.891356    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ae966c-6997-446c-a6b6-52393972b61c-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:36.891441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.891388    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ae966c-6997-446c-a6b6-52393972b61c-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:36.891441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:36.891404    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sffc6\" (UniqueName: \"kubernetes.io/projected/a5ae966c-6997-446c-a6b6-52393972b61c-kube-api-access-sffc6\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:37.457717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:37.457677    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:37.605196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:37.605164    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"
Nov 02 13:32:37.605439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:37.605198    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt" event={"ID":"a5ae966c-6997-446c-a6b6-52393972b61c","Type":"ContainerDied","Data":"6961ec147ce7ca98fd866b2f9e49c2301935f864e837e1950faefd3db4da1922"}
Nov 02 13:32:37.605439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:37.605259    5879 scope.go:122] "RemoveContainer" containerID="61df33f8c78f45658bb0f858fc3360b04eb74f6d53f6411b9487c4b679725daa"
Nov 02 13:32:37.663983 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:37.663949    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"]
Nov 02 13:32:37.677526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:37.677488    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-1aa18c/rate-test-model-1aa18c-kserve-6c9fd7d8d5-j5ddt"]
Nov 02 13:32:38.018842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:38.018805    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ae966c-6997-446c-a6b6-52393972b61c" path="/var/lib/kubelet/pods/a5ae966c-6997-446c-a6b6-52393972b61c/volumes"
Nov 02 13:32:38.208361 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:38.208330    5879 scope.go:122] "RemoveContainer" containerID="ff309e88a243458857fd45985656538da6737bc71a87e6c52cfffcb1f2e866a3"
Nov 02 13:32:38.219250 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:38.219222    5879 scope.go:122] "RemoveContainer" containerID="f620d71a1c1e10c3342cd3b08397f42bab73766aeb08c31b7f3f343f72f482fb"
Nov 02 13:32:38.229497 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:38.229475    5879 scope.go:122] "RemoveContainer" containerID="f3c736a8ed3b033430b2f84012a71605eaaeb8b58f88f6c6a545b8498eeecf6e"
Nov 02 13:32:38.239872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:38.239849    5879 scope.go:122] "RemoveContainer" containerID="88612ea11f1638b6a4777831ac5b4fe989657f9af5c0566099e5c6ba7871df76"
Nov 02 13:32:38.250619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:38.250588    5879 scope.go:122] "RemoveContainer" containerID="de82f29d198ed62e356b58de68539ebe3a88de14c9412adc404c62992d199101"
Nov 02 13:32:39.154654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:39.154616    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:41.089890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:41.089850    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:41.543789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:41.543755    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:42.684079 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:42.684033    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8" podUID="f44570e6-9d69-418d-8a2a-8dcbe46857f2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:32:42.684079 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:42.684041    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn" podUID="8bee752b-e2a7-4348-b924-8615b18822d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:32:42.889927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:42.889876    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:44.273363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.273327    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:44.377572 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.377497    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb" podUID="712a34b5-6d02-4f7b-9f07-ae896cacec19" containerName="main" containerID="cri-o://90072cbd0032a5112d618e94191ae28e815bb71307c87e131c83a74269b207e6" gracePeriod=45
Nov 02 13:32:44.658510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.658463    5879 generic.go:356] "Generic (PLEG): container finished" podID="712a34b5-6d02-4f7b-9f07-ae896cacec19" containerID="90072cbd0032a5112d618e94191ae28e815bb71307c87e131c83a74269b207e6" exitCode=0
Nov 02 13:32:44.658728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.658509    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb" event={"ID":"712a34b5-6d02-4f7b-9f07-ae896cacec19","Type":"ContainerDied","Data":"90072cbd0032a5112d618e94191ae28e815bb71307c87e131c83a74269b207e6"}
Nov 02 13:32:44.697537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.697508    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:32:44.710642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.710608    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-model-cache\") pod \"712a34b5-6d02-4f7b-9f07-ae896cacec19\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") "
Nov 02 13:32:44.710844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.710692    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-dshm\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-dshm\") pod \"712a34b5-6d02-4f7b-9f07-ae896cacec19\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") "
Nov 02 13:32:44.710844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.710713    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-home\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-home\") pod \"712a34b5-6d02-4f7b-9f07-ae896cacec19\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") "
Nov 02 13:32:44.710844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.710768    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-tmp-dir\") pod \"712a34b5-6d02-4f7b-9f07-ae896cacec19\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") "
Nov 02 13:32:44.710844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.710812    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712a34b5-6d02-4f7b-9f07-ae896cacec19-kube-api-access-p496x\" (UniqueName: \"kubernetes.io/projected/712a34b5-6d02-4f7b-9f07-ae896cacec19-kube-api-access-p496x\") pod \"712a34b5-6d02-4f7b-9f07-ae896cacec19\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") "
Nov 02 13:32:44.711133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.710859    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/712a34b5-6d02-4f7b-9f07-ae896cacec19-tls-certs\" (UniqueName: \"kubernetes.io/secret/712a34b5-6d02-4f7b-9f07-ae896cacec19-tls-certs\") pod \"712a34b5-6d02-4f7b-9f07-ae896cacec19\" (UID: \"712a34b5-6d02-4f7b-9f07-ae896cacec19\") "
Nov 02 13:32:44.711133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.710894    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-model-cache" pod "712a34b5-6d02-4f7b-9f07-ae896cacec19" (UID: "712a34b5-6d02-4f7b-9f07-ae896cacec19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:44.711133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.711094    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-tmp-dir" pod "712a34b5-6d02-4f7b-9f07-ae896cacec19" (UID: "712a34b5-6d02-4f7b-9f07-ae896cacec19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:44.711133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.711109    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-home" pod "712a34b5-6d02-4f7b-9f07-ae896cacec19" (UID: "712a34b5-6d02-4f7b-9f07-ae896cacec19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:44.711511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.711489    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:44.711595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.711517    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:44.711595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.711531    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:44.713385 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.713359    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-dshm" pod "712a34b5-6d02-4f7b-9f07-ae896cacec19" (UID: "712a34b5-6d02-4f7b-9f07-ae896cacec19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:32:44.713385 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.713370    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712a34b5-6d02-4f7b-9f07-ae896cacec19-tls-certs" pod "712a34b5-6d02-4f7b-9f07-ae896cacec19" (UID: "712a34b5-6d02-4f7b-9f07-ae896cacec19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:32:44.713548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.713395    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712a34b5-6d02-4f7b-9f07-ae896cacec19-kube-api-access-p496x" pod "712a34b5-6d02-4f7b-9f07-ae896cacec19" (UID: "712a34b5-6d02-4f7b-9f07-ae896cacec19"). InnerVolumeSpecName "kube-api-access-p496x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:44.812642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.812547    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/712a34b5-6d02-4f7b-9f07-ae896cacec19-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:44.812642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.812577    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p496x\" (UniqueName: \"kubernetes.io/projected/712a34b5-6d02-4f7b-9f07-ae896cacec19-kube-api-access-p496x\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:44.812642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.812590    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/712a34b5-6d02-4f7b-9f07-ae896cacec19-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:44.822767 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:32:44.822727    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:32:44.860530 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:44.860492    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:45.666793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:45.666754    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"
Nov 02 13:32:45.666793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:45.666777    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb" event={"ID":"712a34b5-6d02-4f7b-9f07-ae896cacec19","Type":"ContainerDied","Data":"a4fee678f2a2168eed9538512afdd6af23bf8d172ce457131837133cc8f059c7"}
Nov 02 13:32:45.667378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:45.666836    5879 scope.go:122] "RemoveContainer" containerID="90072cbd0032a5112d618e94191ae28e815bb71307c87e131c83a74269b207e6"
Nov 02 13:32:45.743732 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:45.743693    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"]
Nov 02 13:32:45.757363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:45.757328    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-43be82/rate-test-model-43be82-kserve-85c5dc5ff7-v6cqb"]
Nov 02 13:32:46.018201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:46.018115    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712a34b5-6d02-4f7b-9f07-ae896cacec19" path="/var/lib/kubelet/pods/712a34b5-6d02-4f7b-9f07-ae896cacec19/volumes"
Nov 02 13:32:46.484710 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:46.484666    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:48.065644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:48.065610    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:48.543156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:48.543112    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:49.375828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:49.375786    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"]
Nov 02 13:32:49.490275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:49.490234    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:49.889486 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:49.889451    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:51.352768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:51.352731    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:52.683625 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:32:52.683596    5879 logging.go:55] [core] [Channel #18422 SubChannel #18423]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.197:50052", ServerName: "10.134.0.197:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.197:50052: connect: connection refused"
Nov 02 13:32:52.684088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.684066    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"
Nov 02 13:32:52.727930 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.727890    5879 generic.go:356] "Generic (PLEG): container finished" podID="f44570e6-9d69-418d-8a2a-8dcbe46857f2" containerID="a374ab583818a172e51deab2cc508570f1062a60e0236ec7ef6bf10a5d9a16a5" exitCode=0
Nov 02 13:32:52.728178 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.727967    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8" event={"ID":"f44570e6-9d69-418d-8a2a-8dcbe46857f2","Type":"ContainerDied","Data":"a374ab583818a172e51deab2cc508570f1062a60e0236ec7ef6bf10a5d9a16a5"}
Nov 02 13:32:52.730784 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.730746    5879 generic.go:356] "Generic (PLEG): container finished" podID="8bee752b-e2a7-4348-b924-8615b18822d7" containerID="3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796" exitCode=0
Nov 02 13:32:52.731072 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.731047    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn" event={"ID":"8bee752b-e2a7-4348-b924-8615b18822d7","Type":"ContainerDied","Data":"3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796"}
Nov 02 13:32:52.731238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.731222    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn" event={"ID":"8bee752b-e2a7-4348-b924-8615b18822d7","Type":"ContainerDied","Data":"3073934cad934f1ef63add0f903531958558bc907950d1591f7053a3b03950e6"}
Nov 02 13:32:52.731332 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.731251    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"
Nov 02 13:32:52.731815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.731285    5879 scope.go:122] "RemoveContainer" containerID="3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796"
Nov 02 13:32:52.752230 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.752198    5879 scope.go:122] "RemoveContainer" containerID="3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796"
Nov 02 13:32:52.752666 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:32:52.752628    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796\": container with ID starting with 3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796 not found: ID does not exist" containerID="3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796"
Nov 02 13:32:52.752791 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.752672    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796"} err="failed to get container status \"3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796\": rpc error: code = NotFound desc = could not find container \"3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796\": container with ID starting with 3b6a58cef261b442086e9791ba8dedd8982392ad9194a75761af0001d82ce796 not found: ID does not exist"
Nov 02 13:32:52.832808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.832776    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bee752b-e2a7-4348-b924-8615b18822d7-kube-api-access-wfnft\" (UniqueName: \"kubernetes.io/projected/8bee752b-e2a7-4348-b924-8615b18822d7-kube-api-access-wfnft\") pod \"8bee752b-e2a7-4348-b924-8615b18822d7\" (UID: \"8bee752b-e2a7-4348-b924-8615b18822d7\") "
Nov 02 13:32:52.833043 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.832842    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bee752b-e2a7-4348-b924-8615b18822d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bee752b-e2a7-4348-b924-8615b18822d7-plugins-config-volume\") pod \"8bee752b-e2a7-4348-b924-8615b18822d7\" (UID: \"8bee752b-e2a7-4348-b924-8615b18822d7\") "
Nov 02 13:32:52.833366 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.833338    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bee752b-e2a7-4348-b924-8615b18822d7-plugins-config-volume" pod "8bee752b-e2a7-4348-b924-8615b18822d7" (UID: "8bee752b-e2a7-4348-b924-8615b18822d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:52.835330 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.835307    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bee752b-e2a7-4348-b924-8615b18822d7-kube-api-access-wfnft" pod "8bee752b-e2a7-4348-b924-8615b18822d7" (UID: "8bee752b-e2a7-4348-b924-8615b18822d7"). InnerVolumeSpecName "kube-api-access-wfnft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:52.938085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.936589    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfnft\" (UniqueName: \"kubernetes.io/projected/8bee752b-e2a7-4348-b924-8615b18822d7-kube-api-access-wfnft\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:52.938085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.936624    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bee752b-e2a7-4348-b924-8615b18822d7-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:52.938477 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:52.938424    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:53.023430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.023403    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:32:53.095034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.091300    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:53.117901 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.117858    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"]
Nov 02 13:32:53.118144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.117919    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-740f09-56d9b5f5bc-r4xvn"]
Nov 02 13:32:53.138302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.138271    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44570e6-9d69-418d-8a2a-8dcbe46857f2-kube-api-access-plmnr\" (UniqueName: \"kubernetes.io/projected/f44570e6-9d69-418d-8a2a-8dcbe46857f2-kube-api-access-plmnr\") pod \"f44570e6-9d69-418d-8a2a-8dcbe46857f2\" (UID: \"f44570e6-9d69-418d-8a2a-8dcbe46857f2\") "
Nov 02 13:32:53.138480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.138468    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f44570e6-9d69-418d-8a2a-8dcbe46857f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f44570e6-9d69-418d-8a2a-8dcbe46857f2-plugins-config-volume\") pod \"f44570e6-9d69-418d-8a2a-8dcbe46857f2\" (UID: \"f44570e6-9d69-418d-8a2a-8dcbe46857f2\") "
Nov 02 13:32:53.138790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.138768    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44570e6-9d69-418d-8a2a-8dcbe46857f2-plugins-config-volume" pod "f44570e6-9d69-418d-8a2a-8dcbe46857f2" (UID: "f44570e6-9d69-418d-8a2a-8dcbe46857f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:32:53.140590 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.140572    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44570e6-9d69-418d-8a2a-8dcbe46857f2-kube-api-access-plmnr" pod "f44570e6-9d69-418d-8a2a-8dcbe46857f2" (UID: "f44570e6-9d69-418d-8a2a-8dcbe46857f2"). InnerVolumeSpecName "kube-api-access-plmnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:32:53.240398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.240319    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plmnr\" (UniqueName: \"kubernetes.io/projected/f44570e6-9d69-418d-8a2a-8dcbe46857f2-kube-api-access-plmnr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:53.240398 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.240347    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f44570e6-9d69-418d-8a2a-8dcbe46857f2-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:32:53.684295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.684250    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8" podUID="f44570e6-9d69-418d-8a2a-8dcbe46857f2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.197:50052\" within 1s: context deadline exceeded"
Nov 02 13:32:53.684714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.684344    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:32:53.724511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.724475    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:53.742281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.742247    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"
Nov 02 13:32:53.742508 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.742275    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8" event={"ID":"f44570e6-9d69-418d-8a2a-8dcbe46857f2","Type":"ContainerDied","Data":"9ac0c0735c3d1b44288fa89df98c78e01fbc110d25b92d8f9ab7d7c76e560aaf"}
Nov 02 13:32:53.742508 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.742354    5879 scope.go:122] "RemoveContainer" containerID="a374ab583818a172e51deab2cc508570f1062a60e0236ec7ef6bf10a5d9a16a5"
Nov 02 13:32:53.800750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.800707    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"]
Nov 02 13:32:53.814837 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:53.814805    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-740f09-9f78f5f99-v5vz8"]
Nov 02 13:32:54.020538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:54.020459    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bee752b-e2a7-4348-b924-8615b18822d7" path="/var/lib/kubelet/pods/8bee752b-e2a7-4348-b924-8615b18822d7/volumes"
Nov 02 13:32:54.020884 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:54.020871    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f44570e6-9d69-418d-8a2a-8dcbe46857f2" path="/var/lib/kubelet/pods/f44570e6-9d69-418d-8a2a-8dcbe46857f2/volumes"
Nov 02 13:32:54.046910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:54.046880    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:54.781137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:54.781097    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:55.037249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:55.037157    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:56.713839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:56.713801    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:58.077698 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:58.077660    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"]
Nov 02 13:32:58.129361 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:58.129320    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:58.461764 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:58.461716    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:32:58.660254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:32:58.660217    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:00.225692 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:00.225657    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:01.659537 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:01.659499    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"]
Nov 02 13:33:01.695828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:01.695793    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"]
Nov 02 13:33:01.834784 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:01.834754    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:02.042755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:02.042661    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:02.499662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:02.499629    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:03.479186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:03.479146    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:03.748803 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:03.748704    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"]
Nov 02 13:33:03.782634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:03.782603    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"]
Nov 02 13:33:03.914874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:03.914843    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:04.083292 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.083182    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:04.411869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.411825    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm" podUID="c53b600d-86f8-4b1e-84c7-0f3a33b3d53f" containerName="main" containerID="cri-o://aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a" gracePeriod=45
Nov 02 13:33:04.692138 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.692110    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:33:04.800647 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.800614    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-dshm\") pod \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") "
Nov 02 13:33:04.800916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.800710    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-kube-api-access-nn8gm\" (UniqueName: \"kubernetes.io/projected/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-kube-api-access-nn8gm\") pod \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") "
Nov 02 13:33:04.800916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.800753    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tls-certs\") pod \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") "
Nov 02 13:33:04.800916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.800786    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-model-cache\") pod \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") "
Nov 02 13:33:04.800916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.800850    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tmp-dir\") pod \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") "
Nov 02 13:33:04.801238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.801028    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-home\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-home\") pod \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\" (UID: \"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f\") "
Nov 02 13:33:04.801238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.801172    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-model-cache" pod "c53b600d-86f8-4b1e-84c7-0f3a33b3d53f" (UID: "c53b600d-86f8-4b1e-84c7-0f3a33b3d53f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:04.801364 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.801262    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tmp-dir" pod "c53b600d-86f8-4b1e-84c7-0f3a33b3d53f" (UID: "c53b600d-86f8-4b1e-84c7-0f3a33b3d53f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:04.801428 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.801381    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-home" pod "c53b600d-86f8-4b1e-84c7-0f3a33b3d53f" (UID: "c53b600d-86f8-4b1e-84c7-0f3a33b3d53f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:04.801921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.801892    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:04.802114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.801924    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:04.802114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.801941    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:04.803446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.803419    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-dshm" pod "c53b600d-86f8-4b1e-84c7-0f3a33b3d53f" (UID: "c53b600d-86f8-4b1e-84c7-0f3a33b3d53f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:04.803561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.803474    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tls-certs" pod "c53b600d-86f8-4b1e-84c7-0f3a33b3d53f" (UID: "c53b600d-86f8-4b1e-84c7-0f3a33b3d53f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:04.803561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.803484    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-kube-api-access-nn8gm" pod "c53b600d-86f8-4b1e-84c7-0f3a33b3d53f" (UID: "c53b600d-86f8-4b1e-84c7-0f3a33b3d53f"). InnerVolumeSpecName "kube-api-access-nn8gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:04.848923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.848887    5879 generic.go:356] "Generic (PLEG): container finished" podID="c53b600d-86f8-4b1e-84c7-0f3a33b3d53f" containerID="aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a" exitCode=0
Nov 02 13:33:04.849129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.848934    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"
Nov 02 13:33:04.849129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.848968    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm" event={"ID":"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f","Type":"ContainerDied","Data":"aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a"}
Nov 02 13:33:04.849129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.849026    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm" event={"ID":"c53b600d-86f8-4b1e-84c7-0f3a33b3d53f","Type":"ContainerDied","Data":"60da59e159712196b9a9c5b0c04e42bcf28e90eb309f85b93d8024e85c9af25c"}
Nov 02 13:33:04.849129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.849045    5879 scope.go:122] "RemoveContainer" containerID="aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a"
Nov 02 13:33:04.860227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.860172    5879 scope.go:122] "RemoveContainer" containerID="aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a"
Nov 02 13:33:04.860494 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:33:04.860474    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a\": container with ID starting with aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a not found: ID does not exist" containerID="aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a"
Nov 02 13:33:04.860569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.860507    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a"} err="failed to get container status \"aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a\": rpc error: code = NotFound desc = could not find container \"aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a\": container with ID starting with aee2fdb049639ebc428f80a02d7834cf3aeba5019a7c0c5bd214c99e4a59913a not found: ID does not exist"
Nov 02 13:33:04.903501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.903468    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:04.903501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.903498    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn8gm\" (UniqueName: \"kubernetes.io/projected/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-kube-api-access-nn8gm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:04.903501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.903509    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:04.906910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.906878    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"]
Nov 02 13:33:04.910457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:04.910429    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-ed56b9/sub-test-model-ed56b9-kserve-666c86c895-r4vzm"]
Nov 02 13:33:05.297194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:05.297162    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:05.755423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:05.755388    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:06.017713 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:06.017635    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53b600d-86f8-4b1e-84c7-0f3a33b3d53f" path="/var/lib/kubelet/pods/c53b600d-86f8-4b1e-84c7-0f3a33b3d53f/volumes"
Nov 02 13:33:06.660044 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:06.659969    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4" podUID="efdd27e2-57d3-458f-8f9b-ec45183a701d" containerName="payload-processing" containerID="cri-o://dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef" gracePeriod=25
Nov 02 13:33:06.696373 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:06.696310    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk" podUID="c51f6132-b50e-4767-b506-45e1e3a642b5" containerName="payload-pre-processing" containerID="cri-o://d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff" gracePeriod=25
Nov 02 13:33:06.900912 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:06.900880    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:08.246838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:08.246797    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4" podUID="efdd27e2-57d3-458f-8f9b-ec45183a701d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:08.247352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:08.246799    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk" podUID="c51f6132-b50e-4767-b506-45e1e3a642b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:08.563020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:08.562907    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:08.749729 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:08.749660    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z" podUID="5f133de5-ca30-497b-a5b3-8eb19ba650e1" containerName="payload-processing" containerID="cri-o://1f1540949921663576520ae0ceb8273c5d93058af93bef21c0fe8b01b5c88fea" gracePeriod=25
Nov 02 13:33:08.783424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:08.783381    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl" podUID="a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe" containerName="payload-pre-processing" containerID="cri-o://3b145a61967a40558e5fa6c69acf98e3a008e1d067bbd154f17137d759cfeaee" gracePeriod=25
Nov 02 13:33:08.859375 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:08.859289    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:10.098527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:10.098495    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:11.591675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:11.591634    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl" podUID="a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:11.591675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:11.591660    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z" podUID="5f133de5-ca30-497b-a5b3-8eb19ba650e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:11.810665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:11.810628    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:12.262155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:12.262119    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:13.084676 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.084634    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:13.106630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.106560    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv" podUID="c97dbd0f-2281-49dc-8ccf-a9c47da1f9da" containerName="main" containerID="cri-o://cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0" gracePeriod=45
Nov 02 13:33:13.480346 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.480317    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:33:13.647769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.647729    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tls-certs\" (UniqueName: \"kubernetes.io/secret/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tls-certs\") pod \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") "
Nov 02 13:33:13.648030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.647796    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-model-cache\") pod \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") "
Nov 02 13:33:13.648030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.647887    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-dshm\") pod \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") "
Nov 02 13:33:13.648030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.647925    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tmp-dir\") pod \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") "
Nov 02 13:33:13.648266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.648106    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-kube-api-access-4rshr\" (UniqueName: \"kubernetes.io/projected/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-kube-api-access-4rshr\") pod \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") "
Nov 02 13:33:13.648266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.648174    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-home\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-home\") pod \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\" (UID: \"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da\") "
Nov 02 13:33:13.648266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.648183    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tmp-dir" pod "c97dbd0f-2281-49dc-8ccf-a9c47da1f9da" (UID: "c97dbd0f-2281-49dc-8ccf-a9c47da1f9da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:13.648439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.648340    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-model-cache" pod "c97dbd0f-2281-49dc-8ccf-a9c47da1f9da" (UID: "c97dbd0f-2281-49dc-8ccf-a9c47da1f9da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:13.648522 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.648440    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-home" pod "c97dbd0f-2281-49dc-8ccf-a9c47da1f9da" (UID: "c97dbd0f-2281-49dc-8ccf-a9c47da1f9da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:13.648972 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.648945    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:13.649115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.648976    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:13.649115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.649009    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:13.650511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.650486    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-kube-api-access-4rshr" pod "c97dbd0f-2281-49dc-8ccf-a9c47da1f9da" (UID: "c97dbd0f-2281-49dc-8ccf-a9c47da1f9da"). InnerVolumeSpecName "kube-api-access-4rshr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:13.650511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.650503    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tls-certs" pod "c97dbd0f-2281-49dc-8ccf-a9c47da1f9da" (UID: "c97dbd0f-2281-49dc-8ccf-a9c47da1f9da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:33:13.650651 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.650591    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-dshm" pod "c97dbd0f-2281-49dc-8ccf-a9c47da1f9da" (UID: "c97dbd0f-2281-49dc-8ccf-a9c47da1f9da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:33:13.749832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.749801    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:13.749832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.749829    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:13.749832 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.749838    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rshr\" (UniqueName: \"kubernetes.io/projected/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da-kube-api-access-4rshr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:13.894749 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.894716    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:13.924870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.924767    5879 generic.go:356] "Generic (PLEG): container finished" podID="c97dbd0f-2281-49dc-8ccf-a9c47da1f9da" containerID="cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0" exitCode=0
Nov 02 13:33:13.924870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.924815    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"
Nov 02 13:33:13.924870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.924827    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv" event={"ID":"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da","Type":"ContainerDied","Data":"cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0"}
Nov 02 13:33:13.925206 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.924877    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv" event={"ID":"c97dbd0f-2281-49dc-8ccf-a9c47da1f9da","Type":"ContainerDied","Data":"2a9c363a636839dc459bb8f2fd7ca2e41fe090c87ea265f95f788e8eb1deba0b"}
Nov 02 13:33:13.925206 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.924899    5879 scope.go:122] "RemoveContainer" containerID="cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0"
Nov 02 13:33:13.936648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.936620    5879 scope.go:122] "RemoveContainer" containerID="cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0"
Nov 02 13:33:13.936950 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:33:13.936929    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0\": container with ID starting with cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0 not found: ID does not exist" containerID="cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0"
Nov 02 13:33:13.937033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.936959    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0"} err="failed to get container status \"cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0\": rpc error: code = NotFound desc = could not find container \"cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0\": container with ID starting with cc8852b37c96e6654d599841d3cc6777d938193262cf585ef05d1038e94293f0 not found: ID does not exist"
Nov 02 13:33:13.989551 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:13.989501    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"]
Nov 02 13:33:14.001455 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:14.001421    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-1e14d4/sub-test-model-1e14d4-kserve-6cd8797bf9-whffv"]
Nov 02 13:33:14.020232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:14.020191    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c97dbd0f-2281-49dc-8ccf-a9c47da1f9da" path="/var/lib/kubelet/pods/c97dbd0f-2281-49dc-8ccf-a9c47da1f9da/volumes"
Nov 02 13:33:14.822961 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:33:14.822923    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:33:15.534609 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:15.534567    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:15.937501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:15.937466    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:16.517585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:16.517554    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:16.700447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:16.700403    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:17.427867 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:17.427831    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:17.568881 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:17.568834    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:17.926026 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:17.925530    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:18.247071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:18.246921    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk" podUID="c51f6132-b50e-4767-b506-45e1e3a642b5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:18.247071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:18.246929    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4" podUID="efdd27e2-57d3-458f-8f9b-ec45183a701d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:18.597599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:18.597514    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:19.285335 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:19.285300    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:20.675871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:20.675837    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:21.393523 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:21.393487    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:21.591856 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:21.591823    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl" podUID="a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:21.591856 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:21.591848    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z" podUID="5f133de5-ca30-497b-a5b3-8eb19ba650e1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:22.980519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:22.980483    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:24.635472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:24.635436    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:24.774670 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:24.774636    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:24.905872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:24.905771    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:26.141190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:26.141151    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:26.509407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:26.509274    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:26.765109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:26.765024    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:27.021186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.021110    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"
Nov 02 13:33:27.024804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.024779    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"
Nov 02 13:33:27.027055 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.027026    5879 generic.go:356] "Generic (PLEG): container finished" podID="c51f6132-b50e-4767-b506-45e1e3a642b5" containerID="d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff" exitCode=0
Nov 02 13:33:27.027202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.027059    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"
Nov 02 13:33:27.027202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.027110    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk" event={"ID":"c51f6132-b50e-4767-b506-45e1e3a642b5","Type":"ContainerDied","Data":"d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff"}
Nov 02 13:33:27.027202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.027162    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk" event={"ID":"c51f6132-b50e-4767-b506-45e1e3a642b5","Type":"ContainerDied","Data":"4205a6ddc8ae16ddeac7a554c60953e6237ea9c500c29aa2747d09f4931076cb"}
Nov 02 13:33:27.027202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.027189    5879 scope.go:122] "RemoveContainer" containerID="d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff"
Nov 02 13:33:27.028510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.028490    5879 generic.go:356] "Generic (PLEG): container finished" podID="efdd27e2-57d3-458f-8f9b-ec45183a701d" containerID="dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef" exitCode=0
Nov 02 13:33:27.028612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.028535    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"
Nov 02 13:33:27.028612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.028575    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4" event={"ID":"efdd27e2-57d3-458f-8f9b-ec45183a701d","Type":"ContainerDied","Data":"dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef"}
Nov 02 13:33:27.028707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.028627    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4" event={"ID":"efdd27e2-57d3-458f-8f9b-ec45183a701d","Type":"ContainerDied","Data":"28e2263c2c9e621b6cb4a7d8cd5d110c665153cc046f759eb3077c328900c817"}
Nov 02 13:33:27.037619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.037584    5879 scope.go:122] "RemoveContainer" containerID="d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff"
Nov 02 13:33:27.037945 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:33:27.037921    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff\": container with ID starting with d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff not found: ID does not exist" containerID="d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff"
Nov 02 13:33:27.038019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.037962    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff"} err="failed to get container status \"d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff\": rpc error: code = NotFound desc = could not find container \"d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff\": container with ID starting with d26b9ae20026b421152238c741aa3efd8228d2eb3129df5a2f62fb69a479d0ff not found: ID does not exist"
Nov 02 13:33:27.038091 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.037984    5879 scope.go:122] "RemoveContainer" containerID="dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef"
Nov 02 13:33:27.048267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.048246    5879 scope.go:122] "RemoveContainer" containerID="dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef"
Nov 02 13:33:27.048580 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:33:27.048559    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef\": container with ID starting with dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef not found: ID does not exist" containerID="dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef"
Nov 02 13:33:27.048633 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.048588    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef"} err="failed to get container status \"dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef\": rpc error: code = NotFound desc = could not find container \"dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef\": container with ID starting with dc113c3e667704b37dd3ad1d8d0c4353bf7572234d194efebdfd098a2261f9ef not found: ID does not exist"
Nov 02 13:33:27.121563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.121529    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efdd27e2-57d3-458f-8f9b-ec45183a701d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efdd27e2-57d3-458f-8f9b-ec45183a701d-plugins-config-volume\") pod \"efdd27e2-57d3-458f-8f9b-ec45183a701d\" (UID: \"efdd27e2-57d3-458f-8f9b-ec45183a701d\") "
Nov 02 13:33:27.121778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.121595    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efdd27e2-57d3-458f-8f9b-ec45183a701d-kube-api-access-p4r9m\" (UniqueName: \"kubernetes.io/projected/efdd27e2-57d3-458f-8f9b-ec45183a701d-kube-api-access-p4r9m\") pod \"efdd27e2-57d3-458f-8f9b-ec45183a701d\" (UID: \"efdd27e2-57d3-458f-8f9b-ec45183a701d\") "
Nov 02 13:33:27.121778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.121646    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51f6132-b50e-4767-b506-45e1e3a642b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51f6132-b50e-4767-b506-45e1e3a642b5-plugins-config-volume\") pod \"c51f6132-b50e-4767-b506-45e1e3a642b5\" (UID: \"c51f6132-b50e-4767-b506-45e1e3a642b5\") "
Nov 02 13:33:27.121919 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.121841    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51f6132-b50e-4767-b506-45e1e3a642b5-kube-api-access-glrwj\" (UniqueName: \"kubernetes.io/projected/c51f6132-b50e-4767-b506-45e1e3a642b5-kube-api-access-glrwj\") pod \"c51f6132-b50e-4767-b506-45e1e3a642b5\" (UID: \"c51f6132-b50e-4767-b506-45e1e3a642b5\") "
Nov 02 13:33:27.122027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.121979    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efdd27e2-57d3-458f-8f9b-ec45183a701d-plugins-config-volume" pod "efdd27e2-57d3-458f-8f9b-ec45183a701d" (UID: "efdd27e2-57d3-458f-8f9b-ec45183a701d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:27.122155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.122132    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51f6132-b50e-4767-b506-45e1e3a642b5-plugins-config-volume" pod "c51f6132-b50e-4767-b506-45e1e3a642b5" (UID: "c51f6132-b50e-4767-b506-45e1e3a642b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:27.122467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.122443    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efdd27e2-57d3-458f-8f9b-ec45183a701d-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:27.122577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.122472    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c51f6132-b50e-4767-b506-45e1e3a642b5-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:27.124127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.124090    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efdd27e2-57d3-458f-8f9b-ec45183a701d-kube-api-access-p4r9m" pod "efdd27e2-57d3-458f-8f9b-ec45183a701d" (UID: "efdd27e2-57d3-458f-8f9b-ec45183a701d"). InnerVolumeSpecName "kube-api-access-p4r9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:27.124326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.124305    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51f6132-b50e-4767-b506-45e1e3a642b5-kube-api-access-glrwj" pod "c51f6132-b50e-4767-b506-45e1e3a642b5" (UID: "c51f6132-b50e-4767-b506-45e1e3a642b5"). InnerVolumeSpecName "kube-api-access-glrwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:27.223871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.223805    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glrwj\" (UniqueName: \"kubernetes.io/projected/c51f6132-b50e-4767-b506-45e1e3a642b5-kube-api-access-glrwj\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:27.223871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.223845    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4r9m\" (UniqueName: \"kubernetes.io/projected/efdd27e2-57d3-458f-8f9b-ec45183a701d-kube-api-access-p4r9m\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:27.380895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.380860    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"]
Nov 02 13:33:27.386526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.386493    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-90cda9-865d66bfbf-z4rc4"]
Nov 02 13:33:27.423506 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.423468    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"]
Nov 02 13:33:27.431034 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:27.430788    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-90cda9-db6b5d58c-h4psk"]
Nov 02 13:33:28.017982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:28.017898    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51f6132-b50e-4767-b506-45e1e3a642b5" path="/var/lib/kubelet/pods/c51f6132-b50e-4767-b506-45e1e3a642b5/volumes"
Nov 02 13:33:28.018389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:28.018375    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efdd27e2-57d3-458f-8f9b-ec45183a701d" path="/var/lib/kubelet/pods/efdd27e2-57d3-458f-8f9b-ec45183a701d/volumes"
Nov 02 13:33:28.360458 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:28.360362    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:28.501460 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:28.501417    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:28.633756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:28.633718    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:29.050666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.050631    5879 generic.go:356] "Generic (PLEG): container finished" podID="5f133de5-ca30-497b-a5b3-8eb19ba650e1" containerID="1f1540949921663576520ae0ceb8273c5d93058af93bef21c0fe8b01b5c88fea" exitCode=0
Nov 02 13:33:29.050924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.050898    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z" event={"ID":"5f133de5-ca30-497b-a5b3-8eb19ba650e1","Type":"ContainerDied","Data":"1f1540949921663576520ae0ceb8273c5d93058af93bef21c0fe8b01b5c88fea"}
Nov 02 13:33:29.051125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.051105    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z" event={"ID":"5f133de5-ca30-497b-a5b3-8eb19ba650e1","Type":"ContainerDied","Data":"a6325a9c77494253e34d84458a61203134cf769c81ba870142426939c67e994c"}
Nov 02 13:33:29.051321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.051302    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6325a9c77494253e34d84458a61203134cf769c81ba870142426939c67e994c"
Nov 02 13:33:29.053231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.053205    5879 generic.go:356] "Generic (PLEG): container finished" podID="a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe" containerID="3b145a61967a40558e5fa6c69acf98e3a008e1d067bbd154f17137d759cfeaee" exitCode=0
Nov 02 13:33:29.053337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.053261    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl" event={"ID":"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe","Type":"ContainerDied","Data":"3b145a61967a40558e5fa6c69acf98e3a008e1d067bbd154f17137d759cfeaee"}
Nov 02 13:33:29.066320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.066287    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"
Nov 02 13:33:29.113791 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.113763    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"
Nov 02 13:33:29.149977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.149889    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f133de5-ca30-497b-a5b3-8eb19ba650e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f133de5-ca30-497b-a5b3-8eb19ba650e1-plugins-config-volume\") pod \"5f133de5-ca30-497b-a5b3-8eb19ba650e1\" (UID: \"5f133de5-ca30-497b-a5b3-8eb19ba650e1\") "
Nov 02 13:33:29.150235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.150215    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f133de5-ca30-497b-a5b3-8eb19ba650e1-kube-api-access-f2gjl\" (UniqueName: \"kubernetes.io/projected/5f133de5-ca30-497b-a5b3-8eb19ba650e1-kube-api-access-f2gjl\") pod \"5f133de5-ca30-497b-a5b3-8eb19ba650e1\" (UID: \"5f133de5-ca30-497b-a5b3-8eb19ba650e1\") "
Nov 02 13:33:29.150360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.150332    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f133de5-ca30-497b-a5b3-8eb19ba650e1-plugins-config-volume" pod "5f133de5-ca30-497b-a5b3-8eb19ba650e1" (UID: "5f133de5-ca30-497b-a5b3-8eb19ba650e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:29.150722 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.150704    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f133de5-ca30-497b-a5b3-8eb19ba650e1-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:29.152567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.152546    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f133de5-ca30-497b-a5b3-8eb19ba650e1-kube-api-access-f2gjl" pod "5f133de5-ca30-497b-a5b3-8eb19ba650e1" (UID: "5f133de5-ca30-497b-a5b3-8eb19ba650e1"). InnerVolumeSpecName "kube-api-access-f2gjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:29.235563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.235523    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:29.251809 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.251764    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-plugins-config-volume\") pod \"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe\" (UID: \"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe\") "
Nov 02 13:33:29.252069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.251903    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-kube-api-access-2fzcr\" (UniqueName: \"kubernetes.io/projected/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-kube-api-access-2fzcr\") pod \"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe\" (UID: \"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe\") "
Nov 02 13:33:29.252361 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.252328    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-plugins-config-volume" pod "a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe" (UID: "a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:33:29.252643 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.252612    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:29.252708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.252644    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2gjl\" (UniqueName: \"kubernetes.io/projected/5f133de5-ca30-497b-a5b3-8eb19ba650e1-kube-api-access-f2gjl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:29.255023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.254977    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-kube-api-access-2fzcr" pod "a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe" (UID: "a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe"). InnerVolumeSpecName "kube-api-access-2fzcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:33:29.354272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.354230    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fzcr\" (UniqueName: \"kubernetes.io/projected/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe-kube-api-access-2fzcr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:33:29.609111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:29.608982    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:30.062020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:30.061969    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"
Nov 02 13:33:30.062020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:30.061978    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl" event={"ID":"a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe","Type":"ContainerDied","Data":"576ed9ff7d101f3e08fed79a8a4d9587b3fd62fed2cc67734ec1f519330fefaf"}
Nov 02 13:33:30.062248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:30.062101    5879 scope.go:122] "RemoveContainer" containerID="3b145a61967a40558e5fa6c69acf98e3a008e1d067bbd154f17137d759cfeaee"
Nov 02 13:33:30.062248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:30.061986    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"
Nov 02 13:33:30.141455 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:30.141417    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"]
Nov 02 13:33:30.153857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:30.153826    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-25f7a6-796cc7bf6f-5sb8z"]
Nov 02 13:33:30.190532 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:30.190494    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:30.191218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:30.191191    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"]
Nov 02 13:33:30.194092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:30.194062    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-25f7a6-7dc85fc9-47fpl"]
Nov 02 13:33:30.544390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:30.544358    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:31.699267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:31.699196    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:32.017545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:32.017464    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f133de5-ca30-497b-a5b3-8eb19ba650e1" path="/var/lib/kubelet/pods/5f133de5-ca30-497b-a5b3-8eb19ba650e1/volumes"
Nov 02 13:33:32.017910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:32.017896    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe" path="/var/lib/kubelet/pods/a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe/volumes"
Nov 02 13:33:32.469803 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:32.469771    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:32.617091 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:32.617054    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:33.166396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:33.166367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:33.306401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:33.306367    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:33.854168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:33.854127    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:34.370165 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:34.370126    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:34.940237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:34.940208    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:35.139129 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:35.139096    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:36.091164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:36.091131    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:36.216602 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:36.216560    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:36.636653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:36.636614    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:37.002756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:37.002661    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:37.151532 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:37.151492    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"]
Nov 02 13:33:37.190162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:37.190125    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"]
Nov 02 13:33:37.322273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:37.322182    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:37.453232 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:37.453200    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:38.290227 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:38.290195    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:38.321770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:38.321729    5879 scope.go:122] "RemoveContainer" containerID="1f1540949921663576520ae0ceb8273c5d93058af93bef21c0fe8b01b5c88fea"
Nov 02 13:33:39.872861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:39.872828    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:40.317363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:40.317263    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"]
Nov 02 13:33:40.351353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:40.351318    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"]
Nov 02 13:33:40.497045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:40.497007    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:40.627670 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:40.627633    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:41.675857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:41.675815    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:41.815161 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:41.815130    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:41.982745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:41.982661    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:42.152653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:42.152618    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k" podUID="64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d" containerName="payload-processing" containerID="cri-o://f19a5855a6635380af824e664f39ab635f77191a56c77254aae93f4cba4ac52f" gracePeriod=25
Nov 02 13:33:42.190545 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:42.190504    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg" podUID="9a117fc4-daac-490d-b03a-4268ee8aaa5c" containerName="payload-pre-processing" containerID="cri-o://6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b" gracePeriod=25
Nov 02 13:33:43.382214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:43.382170    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:44.589440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:44.589405    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:44.824110 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:33:44.824076    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:33:45.138786 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:45.138738    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:45.318190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:45.318123    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv" podUID="98bb8b0a-8ab1-4150-af7d-42d8429199d7" containerName="payload-processing" containerID="cri-o://0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb" gracePeriod=25
Nov 02 13:33:45.351895 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:45.351849    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx" podUID="3923b227-4f36-4de9-9ef7-b83ea7b28dae" containerName="payload-pre-processing" containerID="cri-o://c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85" gracePeriod=25
Nov 02 13:33:45.457471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:45.457384    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:46.439869 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:46.439828    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:47.058953 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:47.058920    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:48.157969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:48.157933    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k" podUID="64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:48.158443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:48.157935    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg" podUID="9a117fc4-daac-490d-b03a-4268ee8aaa5c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:48.265229 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:48.265195    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:48.387080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:48.387044    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:48.559870 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:48.559788    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:49.693077 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:49.693032    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx" podUID="3923b227-4f36-4de9-9ef7-b83ea7b28dae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:49.693547 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:49.693052    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv" podUID="98bb8b0a-8ab1-4150-af7d-42d8429199d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:50.069704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:50.069595    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:50.954917 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:50.954876    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:51.103072 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:51.103040    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:51.394064 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:51.394032    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:51.640120 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:51.640090    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:52.012148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:52.012109    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:52.169937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:52.169900    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:52.825181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:52.825147    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:53.925635 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:53.925604    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:55.207240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:55.207198    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:55.349693 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:55.349663    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:55.586840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:55.586748    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:55.941515 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:55.941485    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:56.301473 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:56.301389    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:56.767201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:56.767161    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:57.231313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231274    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"]
Nov 02 13:33:57.231838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231825    5879 cpu_manager.go:414] "containerMap: removing container" podUID="efdd27e2-57d3-458f-8f9b-ec45183a701d" containerName="payload-processing"
Nov 02 13:33:57.231886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231840    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efdd27e2-57d3-458f-8f9b-ec45183a701d" containerName="payload-processing"
Nov 02 13:33:57.231886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231853    5879 cpu_manager.go:414] "containerMap: removing container" podUID="712a34b5-6d02-4f7b-9f07-ae896cacec19" containerName="main"
Nov 02 13:33:57.231886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231858    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712a34b5-6d02-4f7b-9f07-ae896cacec19" containerName="main"
Nov 02 13:33:57.231886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231865    5879 cpu_manager.go:414] "containerMap: removing container" podUID="b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed" containerName="main"
Nov 02 13:33:57.231886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231870    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed" containerName="main"
Nov 02 13:33:57.231886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231876    5879 cpu_manager.go:414] "containerMap: removing container" podUID="8bee752b-e2a7-4348-b924-8615b18822d7" containerName="payload-pre-processing"
Nov 02 13:33:57.231886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231880    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bee752b-e2a7-4348-b924-8615b18822d7" containerName="payload-pre-processing"
Nov 02 13:33:57.231886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231889    5879 cpu_manager.go:414] "containerMap: removing container" podUID="8bebb2ec-9564-4e70-9e56-443a8c8f9a8a" containerName="revoke-keys"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231893    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bebb2ec-9564-4e70-9e56-443a8c8f9a8a" containerName="revoke-keys"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231913    5879 cpu_manager.go:414] "containerMap: removing container" podUID="fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231917    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231927    5879 cpu_manager.go:414] "containerMap: removing container" podUID="a5ae966c-6997-446c-a6b6-52393972b61c" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231931    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ae966c-6997-446c-a6b6-52393972b61c" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231946    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c53b600d-86f8-4b1e-84c7-0f3a33b3d53f" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231950    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53b600d-86f8-4b1e-84c7-0f3a33b3d53f" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231959    5879 cpu_manager.go:414] "containerMap: removing container" podUID="a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe" containerName="payload-pre-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231963    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe" containerName="payload-pre-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231979    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f44570e6-9d69-418d-8a2a-8dcbe46857f2" containerName="payload-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.231983    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44570e6-9d69-418d-8a2a-8dcbe46857f2" containerName="payload-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232008    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5f133de5-ca30-497b-a5b3-8eb19ba650e1" containerName="payload-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232012    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f133de5-ca30-497b-a5b3-8eb19ba650e1" containerName="payload-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232018    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c97dbd0f-2281-49dc-8ccf-a9c47da1f9da" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232022    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c97dbd0f-2281-49dc-8ccf-a9c47da1f9da" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232030    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c51f6132-b50e-4767-b506-45e1e3a642b5" containerName="payload-pre-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232034    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51f6132-b50e-4767-b506-45e1e3a642b5" containerName="payload-pre-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232108    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="712a34b5-6d02-4f7b-9f07-ae896cacec19" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232118    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c9b6a2-cc01-445d-9f9f-b9fbcd7776ed" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232124    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bee752b-e2a7-4348-b924-8615b18822d7" containerName="payload-pre-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232129    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bebb2ec-9564-4e70-9e56-443a8c8f9a8a" containerName="revoke-keys"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232137    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe7c00e6-5bde-4c59-bda7-6ea5894cabfb" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232146    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ae966c-6997-446c-a6b6-52393972b61c" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232154    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53b600d-86f8-4b1e-84c7-0f3a33b3d53f" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232159    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3a98fb2-a000-4e7a-ae1e-26d3cea3cdfe" containerName="payload-pre-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232168    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44570e6-9d69-418d-8a2a-8dcbe46857f2" containerName="payload-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232173    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f133de5-ca30-497b-a5b3-8eb19ba650e1" containerName="payload-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232178    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c97dbd0f-2281-49dc-8ccf-a9c47da1f9da" containerName="main"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232183    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51f6132-b50e-4767-b506-45e1e3a642b5" containerName="payload-pre-processing"
Nov 02 13:33:57.232180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.232189    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="efdd27e2-57d3-458f-8f9b-ec45183a701d" containerName="payload-processing"
Nov 02 13:33:57.235179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.235162    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:33:57.254949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.254917    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"]
Nov 02 13:33:57.265470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.265443    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:33:57.265723 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.265703    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:33:57.325134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.325094    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d37229e2-7594z\" (UID: \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:33:57.325599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.325237    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d37229e2-7594z\" (UID: \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:33:57.325599 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.325298    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjml\" (UniqueName: \"kubernetes.io/projected/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-kube-api-access-hqjml\") pod \"maas-api-revoke-sub-d37229e2-7594z\" (UID: \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:33:57.427388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.427344    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d37229e2-7594z\" (UID: \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:33:57.427676 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.427457    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjml\" (UniqueName: \"kubernetes.io/projected/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-kube-api-access-hqjml\") pod \"maas-api-revoke-sub-d37229e2-7594z\" (UID: \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:33:57.427676 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.427655    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d37229e2-7594z\" (UID: \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:33:57.428191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.428163    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d37229e2-7594z\" (UID: \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:33:57.430770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.430737    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d37229e2-7594z\" (UID: \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:33:57.447209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.447177    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:57.457882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.457858    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjml\" (UniqueName: \"kubernetes.io/projected/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-kube-api-access-hqjml\") pod \"maas-api-revoke-sub-d37229e2-7594z\" (UID: \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:33:57.550243 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.550160    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:33:57.857139 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:33:57.857104    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dabe41c_5395_4d96_bf7c_c5dc43b5ed37.slice/crio-6bc9ffbacadfc1f170f35e72127c3b3495b7bf2123557a7e95457166e95e33dc WatchSource:0}: Error finding container 6bc9ffbacadfc1f170f35e72127c3b3495b7bf2123557a7e95457166e95e33dc: Status 404 returned error can't find the container with id 6bc9ffbacadfc1f170f35e72127c3b3495b7bf2123557a7e95457166e95e33dc
Nov 02 13:33:57.858427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:57.858403    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"]
Nov 02 13:33:58.157471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:58.157433    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg" podUID="9a117fc4-daac-490d-b03a-4268ee8aaa5c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:58.157659 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:58.157435    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k" podUID="64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:58.259128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:58.259087    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z" event={"ID":"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37","Type":"ContainerStarted","Data":"bee2a9f738697ac8fe5a4710e4f3b83a66a8be2a29f1aa48ccfa7937fdd5c373"}
Nov 02 13:33:58.259344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:58.259156    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z" event={"ID":"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37","Type":"ContainerStarted","Data":"6bc9ffbacadfc1f170f35e72127c3b3495b7bf2123557a7e95457166e95e33dc"}
Nov 02 13:33:58.284375 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:58.284345    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:58.304106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:58.304052    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z" podStartSLOduration=1.304036967 podStartE2EDuration="1.304036967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:33:58.302427622 +0000 UTC m=+3984.409130473" watchObservedRunningTime="2026-11-02 13:33:58.304036967 +0000 UTC m=+3984.410739817"
Nov 02 13:33:58.425027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:58.424970    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:59.268468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:59.268366    5879 generic.go:356] "Generic (PLEG): container finished" podID="8dabe41c-5395-4d96-bf7c-c5dc43b5ed37" containerID="bee2a9f738697ac8fe5a4710e4f3b83a66a8be2a29f1aa48ccfa7937fdd5c373" exitCode=0
Nov 02 13:33:59.268468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:59.268432    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z" event={"ID":"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37","Type":"ContainerDied","Data":"bee2a9f738697ac8fe5a4710e4f3b83a66a8be2a29f1aa48ccfa7937fdd5c373"}
Nov 02 13:33:59.321018 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:59.320957    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:33:59.692094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:59.692049    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx" podUID="3923b227-4f36-4de9-9ef7-b83ea7b28dae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:59.692094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:59.692085    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv" podUID="98bb8b0a-8ab1-4150-af7d-42d8429199d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:33:59.916012 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:33:59.915963    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:00.425699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:00.425662    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:34:00.570331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:00.570220    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-xks-ca\") pod \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\" (UID: \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\") "
Nov 02 13:34:00.570331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:00.570311    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-kube-api-access-hqjml\" (UniqueName: \"kubernetes.io/projected/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-kube-api-access-hqjml\") pod \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\" (UID: \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\") "
Nov 02 13:34:00.570635 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:00.570464    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-openshift-ca\") pod \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\" (UID: \"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37\") "
Nov 02 13:34:00.570829 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:00.570800    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-openshift-ca" pod "8dabe41c-5395-4d96-bf7c-c5dc43b5ed37" (UID: "8dabe41c-5395-4d96-bf7c-c5dc43b5ed37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:00.572819 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:00.572788    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-xks-ca" pod "8dabe41c-5395-4d96-bf7c-c5dc43b5ed37" (UID: "8dabe41c-5395-4d96-bf7c-c5dc43b5ed37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:00.573048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:00.573021    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-kube-api-access-hqjml" pod "8dabe41c-5395-4d96-bf7c-c5dc43b5ed37" (UID: "8dabe41c-5395-4d96-bf7c-c5dc43b5ed37"). InnerVolumeSpecName "kube-api-access-hqjml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:00.672441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:00.672392    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:00.672441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:00.672434    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:00.672441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:00.672444    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqjml\" (UniqueName: \"kubernetes.io/projected/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37-kube-api-access-hqjml\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:00.947385 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:00.947353    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:01.283216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:01.283129    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"
Nov 02 13:34:01.283216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:01.283161    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z" event={"ID":"8dabe41c-5395-4d96-bf7c-c5dc43b5ed37","Type":"ContainerDied","Data":"6bc9ffbacadfc1f170f35e72127c3b3495b7bf2123557a7e95457166e95e33dc"}
Nov 02 13:34:01.283216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:01.283211    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc9ffbacadfc1f170f35e72127c3b3495b7bf2123557a7e95457166e95e33dc"
Nov 02 13:34:01.846768 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:01.846732    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:02.292496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.292465    5879 generic.go:356] "Generic (PLEG): container finished" podID="64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d" containerID="f19a5855a6635380af824e664f39ab635f77191a56c77254aae93f4cba4ac52f" exitCode=0
Nov 02 13:34:02.292900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.292550    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k" event={"ID":"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d","Type":"ContainerDied","Data":"f19a5855a6635380af824e664f39ab635f77191a56c77254aae93f4cba4ac52f"}
Nov 02 13:34:02.484847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.484819    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"
Nov 02 13:34:02.495234 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.495207    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-kube-api-access-4mbhf\" (UniqueName: \"kubernetes.io/projected/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-kube-api-access-4mbhf\") pod \"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d\" (UID: \"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d\") "
Nov 02 13:34:02.495361 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.495285    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-plugins-config-volume\") pod \"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d\" (UID: \"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d\") "
Nov 02 13:34:02.495665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.495642    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-plugins-config-volume" pod "64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d" (UID: "64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:02.497532 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.497512    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-kube-api-access-4mbhf" pod "64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d" (UID: "64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d"). InnerVolumeSpecName "kube-api-access-4mbhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:02.511439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.511416    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"
Nov 02 13:34:02.547691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.547663    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:02.596092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.596057    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a117fc4-daac-490d-b03a-4268ee8aaa5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a117fc4-daac-490d-b03a-4268ee8aaa5c-plugins-config-volume\") pod \"9a117fc4-daac-490d-b03a-4268ee8aaa5c\" (UID: \"9a117fc4-daac-490d-b03a-4268ee8aaa5c\") "
Nov 02 13:34:02.596299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.596281    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a117fc4-daac-490d-b03a-4268ee8aaa5c-kube-api-access-hd685\" (UniqueName: \"kubernetes.io/projected/9a117fc4-daac-490d-b03a-4268ee8aaa5c-kube-api-access-hd685\") pod \"9a117fc4-daac-490d-b03a-4268ee8aaa5c\" (UID: \"9a117fc4-daac-490d-b03a-4268ee8aaa5c\") "
Nov 02 13:34:02.596814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.596784    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mbhf\" (UniqueName: \"kubernetes.io/projected/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-kube-api-access-4mbhf\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:02.596814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.596814    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:02.597221 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.597149    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a117fc4-daac-490d-b03a-4268ee8aaa5c-plugins-config-volume" pod "9a117fc4-daac-490d-b03a-4268ee8aaa5c" (UID: "9a117fc4-daac-490d-b03a-4268ee8aaa5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:02.600126 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.600102    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a117fc4-daac-490d-b03a-4268ee8aaa5c-kube-api-access-hd685" pod "9a117fc4-daac-490d-b03a-4268ee8aaa5c" (UID: "9a117fc4-daac-490d-b03a-4268ee8aaa5c"). InnerVolumeSpecName "kube-api-access-hd685". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:02.672189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.672092    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:02.698008 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.697958    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd685\" (UniqueName: \"kubernetes.io/projected/9a117fc4-daac-490d-b03a-4268ee8aaa5c-kube-api-access-hd685\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:02.698198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:02.698038    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a117fc4-daac-490d-b03a-4268ee8aaa5c-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:03.300598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.300564    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"
Nov 02 13:34:03.300598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.300580    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k" event={"ID":"64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d","Type":"ContainerDied","Data":"726ecee8f91cdc5b03401c84b513b9c5aec60adfc5e4102e48fafd4949af099e"}
Nov 02 13:34:03.301192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.300645    5879 scope.go:122] "RemoveContainer" containerID="f19a5855a6635380af824e664f39ab635f77191a56c77254aae93f4cba4ac52f"
Nov 02 13:34:03.302363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.302312    5879 generic.go:356] "Generic (PLEG): container finished" podID="9a117fc4-daac-490d-b03a-4268ee8aaa5c" containerID="6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b" exitCode=0
Nov 02 13:34:03.302363 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.302365    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"
Nov 02 13:34:03.302548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.302401    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg" event={"ID":"9a117fc4-daac-490d-b03a-4268ee8aaa5c","Type":"ContainerDied","Data":"6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b"}
Nov 02 13:34:03.302548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.302455    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg" event={"ID":"9a117fc4-daac-490d-b03a-4268ee8aaa5c","Type":"ContainerDied","Data":"1aa7f269dbec979ecc9f0df6ce39604da22e63baaec9aecb5c3441cce5184e17"}
Nov 02 13:34:03.313190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.313137    5879 scope.go:122] "RemoveContainer" containerID="6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b"
Nov 02 13:34:03.323297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.323270    5879 scope.go:122] "RemoveContainer" containerID="6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b"
Nov 02 13:34:03.323621 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:34:03.323601    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b\": container with ID starting with 6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b not found: ID does not exist" containerID="6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b"
Nov 02 13:34:03.323695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.323631    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b"} err="failed to get container status \"6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b\": rpc error: code = NotFound desc = could not find container \"6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b\": container with ID starting with 6cc1257db180b9296e4516e51d51127cdc9e90cfe47b3b6fb4d60f24e2c24b9b not found: ID does not exist"
Nov 02 13:34:03.360121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.360087    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"]
Nov 02 13:34:03.374422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.374386    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0e422b-56ccd9f6c5-l972k"]
Nov 02 13:34:03.403789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.403748    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"]
Nov 02 13:34:03.426048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:03.425919    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0e422b-bd879bb75-hjpqg"]
Nov 02 13:34:04.020191 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:04.020153    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d" path="/var/lib/kubelet/pods/64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d/volumes"
Nov 02 13:34:04.020623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:04.020609    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a117fc4-daac-490d-b03a-4268ee8aaa5c" path="/var/lib/kubelet/pods/9a117fc4-daac-490d-b03a-4268ee8aaa5c/volumes"
Nov 02 13:34:04.053769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:04.053715    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:04.660389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:04.660353    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:04.791872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:04.791834    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:04.992467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:04.992372    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:05.634664 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.634636    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"
Nov 02 13:34:05.664356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.664325    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"
Nov 02 13:34:05.734705 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.734673    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3923b227-4f36-4de9-9ef7-b83ea7b28dae-kube-api-access-kh6sp\" (UniqueName: \"kubernetes.io/projected/3923b227-4f36-4de9-9ef7-b83ea7b28dae-kube-api-access-kh6sp\") pod \"3923b227-4f36-4de9-9ef7-b83ea7b28dae\" (UID: \"3923b227-4f36-4de9-9ef7-b83ea7b28dae\") "
Nov 02 13:34:05.734705 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.734718    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98bb8b0a-8ab1-4150-af7d-42d8429199d7-kube-api-access-df2rl\" (UniqueName: \"kubernetes.io/projected/98bb8b0a-8ab1-4150-af7d-42d8429199d7-kube-api-access-df2rl\") pod \"98bb8b0a-8ab1-4150-af7d-42d8429199d7\" (UID: \"98bb8b0a-8ab1-4150-af7d-42d8429199d7\") "
Nov 02 13:34:05.734952 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.734792    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3923b227-4f36-4de9-9ef7-b83ea7b28dae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3923b227-4f36-4de9-9ef7-b83ea7b28dae-plugins-config-volume\") pod \"3923b227-4f36-4de9-9ef7-b83ea7b28dae\" (UID: \"3923b227-4f36-4de9-9ef7-b83ea7b28dae\") "
Nov 02 13:34:05.734952 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.734876    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98bb8b0a-8ab1-4150-af7d-42d8429199d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bb8b0a-8ab1-4150-af7d-42d8429199d7-plugins-config-volume\") pod \"98bb8b0a-8ab1-4150-af7d-42d8429199d7\" (UID: \"98bb8b0a-8ab1-4150-af7d-42d8429199d7\") "
Nov 02 13:34:05.735329 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.735298    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3923b227-4f36-4de9-9ef7-b83ea7b28dae-plugins-config-volume" pod "3923b227-4f36-4de9-9ef7-b83ea7b28dae" (UID: "3923b227-4f36-4de9-9ef7-b83ea7b28dae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:05.735329 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.735310    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98bb8b0a-8ab1-4150-af7d-42d8429199d7-plugins-config-volume" pod "98bb8b0a-8ab1-4150-af7d-42d8429199d7" (UID: "98bb8b0a-8ab1-4150-af7d-42d8429199d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:05.738108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.735430    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3923b227-4f36-4de9-9ef7-b83ea7b28dae-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:05.738108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.735452    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98bb8b0a-8ab1-4150-af7d-42d8429199d7-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:05.738108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.737470    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3923b227-4f36-4de9-9ef7-b83ea7b28dae-kube-api-access-kh6sp" pod "3923b227-4f36-4de9-9ef7-b83ea7b28dae" (UID: "3923b227-4f36-4de9-9ef7-b83ea7b28dae"). InnerVolumeSpecName "kube-api-access-kh6sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:05.738108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.737501    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bb8b0a-8ab1-4150-af7d-42d8429199d7-kube-api-access-df2rl" pod "98bb8b0a-8ab1-4150-af7d-42d8429199d7" (UID: "98bb8b0a-8ab1-4150-af7d-42d8429199d7"). InnerVolumeSpecName "kube-api-access-df2rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:05.837147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.837050    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh6sp\" (UniqueName: \"kubernetes.io/projected/3923b227-4f36-4de9-9ef7-b83ea7b28dae-kube-api-access-kh6sp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:05.837147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:05.837089    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df2rl\" (UniqueName: \"kubernetes.io/projected/98bb8b0a-8ab1-4150-af7d-42d8429199d7-kube-api-access-df2rl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:06.327448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.327408    5879 generic.go:356] "Generic (PLEG): container finished" podID="3923b227-4f36-4de9-9ef7-b83ea7b28dae" containerID="c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85" exitCode=0
Nov 02 13:34:06.327679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.327463    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"
Nov 02 13:34:06.327679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.327492    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx" event={"ID":"3923b227-4f36-4de9-9ef7-b83ea7b28dae","Type":"ContainerDied","Data":"c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85"}
Nov 02 13:34:06.327679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.327554    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx" event={"ID":"3923b227-4f36-4de9-9ef7-b83ea7b28dae","Type":"ContainerDied","Data":"bf9da386f77f55852996a1b9b59aa8b9a0bfb70ce16bb4bbc46f88c3ba6a4174"}
Nov 02 13:34:06.327679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.327582    5879 scope.go:122] "RemoveContainer" containerID="c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85"
Nov 02 13:34:06.329252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.329229    5879 generic.go:356] "Generic (PLEG): container finished" podID="98bb8b0a-8ab1-4150-af7d-42d8429199d7" containerID="0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb" exitCode=0
Nov 02 13:34:06.329401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.329270    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"
Nov 02 13:34:06.329401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.329313    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv" event={"ID":"98bb8b0a-8ab1-4150-af7d-42d8429199d7","Type":"ContainerDied","Data":"0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb"}
Nov 02 13:34:06.329401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.329351    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv" event={"ID":"98bb8b0a-8ab1-4150-af7d-42d8429199d7","Type":"ContainerDied","Data":"02cd7e23debad1663a4b39868640675eaccc6581914582d50f03f43a7843622b"}
Nov 02 13:34:06.347266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.347238    5879 scope.go:122] "RemoveContainer" containerID="c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85"
Nov 02 13:34:06.347644 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:34:06.347618    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85\": container with ID starting with c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85 not found: ID does not exist" containerID="c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85"
Nov 02 13:34:06.347736 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.347652    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85"} err="failed to get container status \"c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85\": rpc error: code = NotFound desc = could not find container \"c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85\": container with ID starting with c52854b6934bba3feb59415b47084641af61d058c95e25fea3c88eca41b83e85 not found: ID does not exist"
Nov 02 13:34:06.347736 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.347670    5879 scope.go:122] "RemoveContainer" containerID="0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb"
Nov 02 13:34:06.357860 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.357840    5879 scope.go:122] "RemoveContainer" containerID="0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb"
Nov 02 13:34:06.358159 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:34:06.358137    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb\": container with ID starting with 0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb not found: ID does not exist" containerID="0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb"
Nov 02 13:34:06.358218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.358172    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb"} err="failed to get container status \"0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb\": rpc error: code = NotFound desc = could not find container \"0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb\": container with ID starting with 0b5ba21fee0e8bfcfde0bc8bc8446ca3f180ba7785a6be01aa967251f99a0ebb not found: ID does not exist"
Nov 02 13:34:06.395276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.395240    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"]
Nov 02 13:34:06.409305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.409266    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-1aa18c-7dccf58c79-vtkmx"]
Nov 02 13:34:06.437935 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.437902    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"]
Nov 02 13:34:06.440521 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.440500    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-1aa18c-7645874b6c-4vflv"]
Nov 02 13:34:06.510365 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.510327    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:06.866509 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.866468    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:06.990965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:06.990931    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:07.534133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:07.534099    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:07.926811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:07.926775    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:08.021606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.021575    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3923b227-4f36-4de9-9ef7-b83ea7b28dae" path="/var/lib/kubelet/pods/3923b227-4f36-4de9-9ef7-b83ea7b28dae/volumes"
Nov 02 13:34:08.022300 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.022277    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98bb8b0a-8ab1-4150-af7d-42d8429199d7" path="/var/lib/kubelet/pods/98bb8b0a-8ab1-4150-af7d-42d8429199d7/volumes"
Nov 02 13:34:08.110850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.110813    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"]
Nov 02 13:34:08.111482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111462    5879 cpu_manager.go:414] "containerMap: removing container" podUID="3923b227-4f36-4de9-9ef7-b83ea7b28dae" containerName="payload-pre-processing"
Nov 02 13:34:08.111482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111482    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3923b227-4f36-4de9-9ef7-b83ea7b28dae" containerName="payload-pre-processing"
Nov 02 13:34:08.111662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111505    5879 cpu_manager.go:414] "containerMap: removing container" podUID="9a117fc4-daac-490d-b03a-4268ee8aaa5c" containerName="payload-pre-processing"
Nov 02 13:34:08.111662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111514    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a117fc4-daac-490d-b03a-4268ee8aaa5c" containerName="payload-pre-processing"
Nov 02 13:34:08.111662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111529    5879 cpu_manager.go:414] "containerMap: removing container" podUID="98bb8b0a-8ab1-4150-af7d-42d8429199d7" containerName="payload-processing"
Nov 02 13:34:08.111662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111534    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98bb8b0a-8ab1-4150-af7d-42d8429199d7" containerName="payload-processing"
Nov 02 13:34:08.111662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111551    5879 cpu_manager.go:414] "containerMap: removing container" podUID="8dabe41c-5395-4d96-bf7c-c5dc43b5ed37" containerName="revoke-keys"
Nov 02 13:34:08.111662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111555    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dabe41c-5395-4d96-bf7c-c5dc43b5ed37" containerName="revoke-keys"
Nov 02 13:34:08.111662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111574    5879 cpu_manager.go:414] "containerMap: removing container" podUID="64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d" containerName="payload-processing"
Nov 02 13:34:08.111662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111578    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d" containerName="payload-processing"
Nov 02 13:34:08.111662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111662    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="3923b227-4f36-4de9-9ef7-b83ea7b28dae" containerName="payload-pre-processing"
Nov 02 13:34:08.112178 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111671    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a117fc4-daac-490d-b03a-4268ee8aaa5c" containerName="payload-pre-processing"
Nov 02 13:34:08.112178 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111678    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="98bb8b0a-8ab1-4150-af7d-42d8429199d7" containerName="payload-processing"
Nov 02 13:34:08.112178 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111685    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dabe41c-5395-4d96-bf7c-c5dc43b5ed37" containerName="revoke-keys"
Nov 02 13:34:08.112178 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.111696    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f7ac6a-9176-4d3e-9dee-2ec9abb2b68d" containerName="payload-processing"
Nov 02 13:34:08.116658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.116634    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:08.134287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.134256    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"]
Nov 02 13:34:08.144618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.144588    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:34:08.144842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.144677    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:34:08.168707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.168645    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f965e06-qkfmq\" (UID: \"72323342-920f-4310-8cb8-9a0c558b285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:08.168922 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.168881    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f965e06-qkfmq\" (UID: \"72323342-920f-4310-8cb8-9a0c558b285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:08.169020 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.168956    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c556x\" (UniqueName: \"kubernetes.io/projected/72323342-920f-4310-8cb8-9a0c558b285a-kube-api-access-c556x\") pod \"maas-api-revoke-sub-5f965e06-qkfmq\" (UID: \"72323342-920f-4310-8cb8-9a0c558b285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:08.270380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.270283    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f965e06-qkfmq\" (UID: \"72323342-920f-4310-8cb8-9a0c558b285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:08.270380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.270346    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c556x\" (UniqueName: \"kubernetes.io/projected/72323342-920f-4310-8cb8-9a0c558b285a-kube-api-access-c556x\") pod \"maas-api-revoke-sub-5f965e06-qkfmq\" (UID: \"72323342-920f-4310-8cb8-9a0c558b285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:08.270633 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.270503    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f965e06-qkfmq\" (UID: \"72323342-920f-4310-8cb8-9a0c558b285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:08.271183 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.271160    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f965e06-qkfmq\" (UID: \"72323342-920f-4310-8cb8-9a0c558b285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:08.273061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.273042    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f965e06-qkfmq\" (UID: \"72323342-920f-4310-8cb8-9a0c558b285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:08.295733 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.295702    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c556x\" (UniqueName: \"kubernetes.io/projected/72323342-920f-4310-8cb8-9a0c558b285a-kube-api-access-c556x\") pod \"maas-api-revoke-sub-5f965e06-qkfmq\" (UID: \"72323342-920f-4310-8cb8-9a0c558b285a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:08.297498 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.297477    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:08.432710 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.432672    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:08.679395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.679364    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:08.741982 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:34:08.741953    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72323342_920f_4310_8cb8_9a0c558b285a.slice/crio-b4b1aa78967a6e9c3687feec5c5fcc44cb8b3b6ed9febe1cfce352dc4bc5b0b7 WatchSource:0}: Error finding container b4b1aa78967a6e9c3687feec5c5fcc44cb8b3b6ed9febe1cfce352dc4bc5b0b7: Status 404 returned error can't find the container with id b4b1aa78967a6e9c3687feec5c5fcc44cb8b3b6ed9febe1cfce352dc4bc5b0b7
Nov 02 13:34:08.742891 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:08.742862    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"]
Nov 02 13:34:09.356287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:09.356198    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq" event={"ID":"72323342-920f-4310-8cb8-9a0c558b285a","Type":"ContainerStarted","Data":"da21ac5db4d48b334176d6673e961ac92a99e1a557cf92099ece153527f5b844"}
Nov 02 13:34:09.356287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:09.356262    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq" event={"ID":"72323342-920f-4310-8cb8-9a0c558b285a","Type":"ContainerStarted","Data":"b4b1aa78967a6e9c3687feec5c5fcc44cb8b3b6ed9febe1cfce352dc4bc5b0b7"}
Nov 02 13:34:09.362540 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:09.362505    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:09.401430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:09.401367    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq" podStartSLOduration=1.401347077 podStartE2EDuration="1.401347077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:34:09.400398758 +0000 UTC m=+3995.507101614" watchObservedRunningTime="2026-11-02 13:34:09.401347077 +0000 UTC m=+3995.508049933"
Nov 02 13:34:09.494697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:09.494666    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:10.109780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:10.109684    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:10.364115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:10.364086    5879 generic.go:356] "Generic (PLEG): container finished" podID="72323342-920f-4310-8cb8-9a0c558b285a" containerID="da21ac5db4d48b334176d6673e961ac92a99e1a557cf92099ece153527f5b844" exitCode=0
Nov 02 13:34:10.364520 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:10.364180    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq" event={"ID":"72323342-920f-4310-8cb8-9a0c558b285a","Type":"ContainerDied","Data":"da21ac5db4d48b334176d6673e961ac92a99e1a557cf92099ece153527f5b844"}
Nov 02 13:34:11.522179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:11.522143    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:11.633974 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:11.633885    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72323342-920f-4310-8cb8-9a0c558b285a-kube-api-access-c556x\" (UniqueName: \"kubernetes.io/projected/72323342-920f-4310-8cb8-9a0c558b285a-kube-api-access-c556x\") pod \"72323342-920f-4310-8cb8-9a0c558b285a\" (UID: \"72323342-920f-4310-8cb8-9a0c558b285a\") "
Nov 02 13:34:11.633974 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:11.633926    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-xks-ca\") pod \"72323342-920f-4310-8cb8-9a0c558b285a\" (UID: \"72323342-920f-4310-8cb8-9a0c558b285a\") "
Nov 02 13:34:11.634210 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:11.634066    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-openshift-ca\") pod \"72323342-920f-4310-8cb8-9a0c558b285a\" (UID: \"72323342-920f-4310-8cb8-9a0c558b285a\") "
Nov 02 13:34:11.634444 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:11.634421    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-openshift-ca" pod "72323342-920f-4310-8cb8-9a0c558b285a" (UID: "72323342-920f-4310-8cb8-9a0c558b285a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:11.636431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:11.636404    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-xks-ca" pod "72323342-920f-4310-8cb8-9a0c558b285a" (UID: "72323342-920f-4310-8cb8-9a0c558b285a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:11.636538 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:11.636433    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72323342-920f-4310-8cb8-9a0c558b285a-kube-api-access-c556x" pod "72323342-920f-4310-8cb8-9a0c558b285a" (UID: "72323342-920f-4310-8cb8-9a0c558b285a"). InnerVolumeSpecName "kube-api-access-c556x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:11.735404 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:11.735365    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:11.735404 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:11.735402    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c556x\" (UniqueName: \"kubernetes.io/projected/72323342-920f-4310-8cb8-9a0c558b285a-kube-api-access-c556x\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:11.735627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:11.735416    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72323342-920f-4310-8cb8-9a0c558b285a-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:11.756708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:11.756675    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:12.379543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:12.379506    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"
Nov 02 13:34:12.379543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:12.379518    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq" event={"ID":"72323342-920f-4310-8cb8-9a0c558b285a","Type":"ContainerDied","Data":"b4b1aa78967a6e9c3687feec5c5fcc44cb8b3b6ed9febe1cfce352dc4bc5b0b7"}
Nov 02 13:34:12.379785 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:12.379578    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4b1aa78967a6e9c3687feec5c5fcc44cb8b3b6ed9febe1cfce352dc4bc5b0b7"
Nov 02 13:34:13.454309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:13.454267    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:14.622367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:14.622330    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"]
Nov 02 13:34:14.659519 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:14.659484    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"]
Nov 02 13:34:14.802021 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:14.801965    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:14.822911 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:34:14.822874    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:34:14.941818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:14.941733    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:15.212175 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:15.212084    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:15.234791 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:15.234754    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"]
Nov 02 13:34:15.235066 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:15.235006    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" podUID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" containerName="istio-proxy" containerID="cri-o://1a7ef25a8ee527b72dc999798796907b2c3a0fd8a38eda96255ec0688400aead" gracePeriod=30
Nov 02 13:34:15.774330 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:15.774299    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:16.035346 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:16.035258    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:16.894607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:16.894576    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:18.840810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:18.840777    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:19.623619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:19.623581    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8" podUID="63f632d1-51bb-4f53-b7eb-afc0dff0cce2" containerName="payload-processing" containerID="cri-o://8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd" gracePeriod=25
Nov 02 13:34:19.660410 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:19.660366    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz" podUID="0c33d267-6b2a-41e0-ba1c-685284c7137e" containerName="payload-pre-processing" containerID="cri-o://40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146" gracePeriod=25
Nov 02 13:34:20.206293 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.206253    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:20.438111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.437742    5879 generic.go:356] "Generic (PLEG): container finished" podID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" containerID="1a7ef25a8ee527b72dc999798796907b2c3a0fd8a38eda96255ec0688400aead" exitCode=0
Nov 02 13:34:20.438111 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.437966    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" event={"ID":"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc","Type":"ContainerDied","Data":"1a7ef25a8ee527b72dc999798796907b2c3a0fd8a38eda96255ec0688400aead"}
Nov 02 13:34:20.580791 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.580763    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:34:20.754133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754097    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-socket\") pod \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") "
Nov 02 13:34:20.754326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754177    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istiod-ca-cert\") pod \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") "
Nov 02 13:34:20.754326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754209    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-credential-socket\") pod \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") "
Nov 02 13:34:20.754326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754241    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-certs\") pod \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") "
Nov 02 13:34:20.754326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754269    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-envoy\") pod \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") "
Nov 02 13:34:20.754326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754301    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-podinfo\") pod \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") "
Nov 02 13:34:20.754616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754398    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-socket" pod "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" (UID: "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:20.754616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754421    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-token\" (UniqueName: \"kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-token\") pod \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") "
Nov 02 13:34:20.754616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754484    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-kube-api-access-7sc5r\" (UniqueName: \"kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-kube-api-access-7sc5r\") pod \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") "
Nov 02 13:34:20.754616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754545    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-credential-socket" pod "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" (UID: "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:20.754616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754601    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-data\") pod \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\" (UID: \"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc\") "
Nov 02 13:34:20.754979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754618    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istiod-ca-cert" pod "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" (UID: "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:20.754979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754846    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-certs" pod "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" (UID: "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:20.754979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.754879    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-data" pod "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" (UID: "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:20.758894 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.758867    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:20.758894 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.758895    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:20.759207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.758908    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:20.759207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.758921    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:20.759207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.758933    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:20.759863 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.759834    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-kube-api-access-7sc5r" pod "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" (UID: "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc"). InnerVolumeSpecName "kube-api-access-7sc5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:20.759987 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.759880    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-token" pod "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" (UID: "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:20.760099 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.760027    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-podinfo" pod "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" (UID: "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:34:20.760269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.760248    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-envoy" pod "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" (UID: "0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:20.851310 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.851218    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:20.860648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.860621    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sc5r\" (UniqueName: \"kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-kube-api-access-7sc5r\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:20.860787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.860651    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:20.860787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.860663    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:20.860787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:20.860672    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:21.446521 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:21.446487    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"
Nov 02 13:34:21.446960 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:21.446516    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb" event={"ID":"0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc","Type":"ContainerDied","Data":"a4d9a75deb2d0fcb9ba68ab7943e928bc488709a3594c3816f331f0c2acbc8d9"}
Nov 02 13:34:21.446960 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:21.446583    5879 scope.go:122] "RemoveContainer" containerID="1a7ef25a8ee527b72dc999798796907b2c3a0fd8a38eda96255ec0688400aead"
Nov 02 13:34:21.518820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:21.518784    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"]
Nov 02 13:34:21.532773 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:21.532732    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-4ncwb"]
Nov 02 13:34:21.986259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:21.986225    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:22.017176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:22.017147    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" path="/var/lib/kubelet/pods/0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc/volumes"
Nov 02 13:34:22.436890 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:22.436853    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:22.782621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:22.782532    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:22.948465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:22.948420    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:23.734810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:23.734764    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8" podUID="63f632d1-51bb-4f53-b7eb-afc0dff0cce2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:23.734979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:23.734807    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz" podUID="0c33d267-6b2a-41e0-ba1c-685284c7137e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:24.060797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:24.060701    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:25.545728 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:25.545681    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:26.177587 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:26.177550    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:27.719931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:27.719900    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:28.348631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:28.348590    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:28.487199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:28.487153    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:29.061392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:29.061365    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:29.357476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:29.357341    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:29.490714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:29.490680    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:29.891077 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:29.891043    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:31.073842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:31.073810    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:32.744023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:32.743920    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:33.734204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:33.734168    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8" podUID="63f632d1-51bb-4f53-b7eb-afc0dff0cce2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:33.734396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:33.734224    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz" podUID="0c33d267-6b2a-41e0-ba1c-685284c7137e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:34:34.339010 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:34.338967    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:34.829595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:34.829558    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"]
Nov 02 13:34:34.834834 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:34.834803    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"]
Nov 02 13:34:34.848201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:34.848165    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"]
Nov 02 13:34:34.974112 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:34.974072    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:35.700139 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:35.700101    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:36.111940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:36.111825    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:37.929204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:37.929170    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:39.447813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:39.447767    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:39.596466 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:39.596422    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:39.816396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:39.816365    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"
Nov 02 13:34:39.913441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:39.913378    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:39.937402 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:39.937377    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"
Nov 02 13:34:39.984329 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:39.984236    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-plugins-config-volume\") pod \"63f632d1-51bb-4f53-b7eb-afc0dff0cce2\" (UID: \"63f632d1-51bb-4f53-b7eb-afc0dff0cce2\") "
Nov 02 13:34:39.984329 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:39.984315    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-kube-api-access-dgw5v\" (UniqueName: \"kubernetes.io/projected/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-kube-api-access-dgw5v\") pod \"63f632d1-51bb-4f53-b7eb-afc0dff0cce2\" (UID: \"63f632d1-51bb-4f53-b7eb-afc0dff0cce2\") "
Nov 02 13:34:39.984633 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:39.984601    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-plugins-config-volume" pod "63f632d1-51bb-4f53-b7eb-afc0dff0cce2" (UID: "63f632d1-51bb-4f53-b7eb-afc0dff0cce2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:39.984730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:39.984715    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:39.986788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:39.986759    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-kube-api-access-dgw5v" pod "63f632d1-51bb-4f53-b7eb-afc0dff0cce2" (UID: "63f632d1-51bb-4f53-b7eb-afc0dff0cce2"). InnerVolumeSpecName "kube-api-access-dgw5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:40.085158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.085121    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c33d267-6b2a-41e0-ba1c-685284c7137e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c33d267-6b2a-41e0-ba1c-685284c7137e-plugins-config-volume\") pod \"0c33d267-6b2a-41e0-ba1c-685284c7137e\" (UID: \"0c33d267-6b2a-41e0-ba1c-685284c7137e\") "
Nov 02 13:34:40.085389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.085230    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c33d267-6b2a-41e0-ba1c-685284c7137e-kube-api-access-lr2zr\" (UniqueName: \"kubernetes.io/projected/0c33d267-6b2a-41e0-ba1c-685284c7137e-kube-api-access-lr2zr\") pod \"0c33d267-6b2a-41e0-ba1c-685284c7137e\" (UID: \"0c33d267-6b2a-41e0-ba1c-685284c7137e\") "
Nov 02 13:34:40.086268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.085657    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c33d267-6b2a-41e0-ba1c-685284c7137e-plugins-config-volume" pod "0c33d267-6b2a-41e0-ba1c-685284c7137e" (UID: "0c33d267-6b2a-41e0-ba1c-685284c7137e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:40.086268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.085955    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c33d267-6b2a-41e0-ba1c-685284c7137e-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:40.086268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.085973    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgw5v\" (UniqueName: \"kubernetes.io/projected/63f632d1-51bb-4f53-b7eb-afc0dff0cce2-kube-api-access-dgw5v\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:40.088531 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.088501    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c33d267-6b2a-41e0-ba1c-685284c7137e-kube-api-access-lr2zr" pod "0c33d267-6b2a-41e0-ba1c-685284c7137e" (UID: "0c33d267-6b2a-41e0-ba1c-685284c7137e"). InnerVolumeSpecName "kube-api-access-lr2zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:40.187862 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.187828    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr2zr\" (UniqueName: \"kubernetes.io/projected/0c33d267-6b2a-41e0-ba1c-685284c7137e-kube-api-access-lr2zr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:40.600309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.600262    5879 generic.go:356] "Generic (PLEG): container finished" podID="63f632d1-51bb-4f53-b7eb-afc0dff0cce2" containerID="8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd" exitCode=0
Nov 02 13:34:40.600959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.600334    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"
Nov 02 13:34:40.600959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.600358    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8" event={"ID":"63f632d1-51bb-4f53-b7eb-afc0dff0cce2","Type":"ContainerDied","Data":"8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd"}
Nov 02 13:34:40.600959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.600424    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8" event={"ID":"63f632d1-51bb-4f53-b7eb-afc0dff0cce2","Type":"ContainerDied","Data":"ba81f34a06d513da2314691ed78dafd0863caad9aa8332e803b0f739e5b541e0"}
Nov 02 13:34:40.600959 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.600456    5879 scope.go:122] "RemoveContainer" containerID="8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd"
Nov 02 13:34:40.620873 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.620837    5879 generic.go:356] "Generic (PLEG): container finished" podID="0c33d267-6b2a-41e0-ba1c-685284c7137e" containerID="40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146" exitCode=0
Nov 02 13:34:40.621068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.620963    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz" event={"ID":"0c33d267-6b2a-41e0-ba1c-685284c7137e","Type":"ContainerDied","Data":"40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146"}
Nov 02 13:34:40.621068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.621032    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz" event={"ID":"0c33d267-6b2a-41e0-ba1c-685284c7137e","Type":"ContainerDied","Data":"c9ca2c9e58154790fc0c6c9da27dcb8e23849f7c040907863d44126149680ac4"}
Nov 02 13:34:40.621250 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.621140    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"
Nov 02 13:34:40.634876 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.634843    5879 scope.go:122] "RemoveContainer" containerID="8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd"
Nov 02 13:34:40.635319 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:34:40.635294    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd\": container with ID starting with 8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd not found: ID does not exist" containerID="8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd"
Nov 02 13:34:40.635432 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.635331    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd"} err="failed to get container status \"8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd\": rpc error: code = NotFound desc = could not find container \"8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd\": container with ID starting with 8e76c57d7d70e12722803ae626a49cff0d626be3c9756ab12df1367771d323bd not found: ID does not exist"
Nov 02 13:34:40.635432 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.635354    5879 scope.go:122] "RemoveContainer" containerID="40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146"
Nov 02 13:34:40.650664 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.650644    5879 scope.go:122] "RemoveContainer" containerID="40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146"
Nov 02 13:34:40.651036 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:34:40.651006    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146\": container with ID starting with 40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146 not found: ID does not exist" containerID="40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146"
Nov 02 13:34:40.651176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.651042    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146"} err="failed to get container status \"40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146\": rpc error: code = NotFound desc = could not find container \"40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146\": container with ID starting with 40f159214de9d1996ece0cfeea7cdb35455693a3c921d189eab2e09878fba146 not found: ID does not exist"
Nov 02 13:34:40.688578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.688531    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"]
Nov 02 13:34:40.695846 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.695812    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:40.702464 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.702437    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-1e14d4-5f947fcf8f-blxl8"]
Nov 02 13:34:40.765449 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.765415    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"]
Nov 02 13:34:40.768241 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:40.768215    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-1e14d4-5b9c594759-v2rpz"]
Nov 02 13:34:41.785581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:41.785536    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:41.933769 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:41.933728    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:42.026016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:42.024197    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c33d267-6b2a-41e0-ba1c-685284c7137e" path="/var/lib/kubelet/pods/0c33d267-6b2a-41e0-ba1c-685284c7137e/volumes"
Nov 02 13:34:42.026016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:42.024909    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f632d1-51bb-4f53-b7eb-afc0dff0cce2" path="/var/lib/kubelet/pods/63f632d1-51bb-4f53-b7eb-afc0dff0cce2/volumes"
Nov 02 13:34:43.003468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:43.003418    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:43.349787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:43.349656    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"]
Nov 02 13:34:43.350905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:43.350872    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy" containerID="cri-o://a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983" gracePeriod=30
Nov 02 13:34:43.431489 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:43.431450    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:43.562946 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:43.562894    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:43.687637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:43.687603    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:43.815763 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:43.815726    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:44.822375 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:34:44.822337    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:34:45.386533 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:45.386498    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:45.700313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:45.700229    5879 patch_prober.go:29] interesting pod/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 13:34:45.700313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:45.700293    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 13:34:46.894277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:46.894242    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:47.066097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:47.066065    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:48.630446 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.630373    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 13:34:48.676818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.676781    5879 generic.go:356] "Generic (PLEG): container finished" podID="960e55b4-4d31-4210-b756-28c039d88f7f" containerID="a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983" exitCode=0
Nov 02 13:34:48.677019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.676833    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"
Nov 02 13:34:48.677019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.676834    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" event={"ID":"960e55b4-4d31-4210-b756-28c039d88f7f","Type":"ContainerDied","Data":"a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983"}
Nov 02 13:34:48.677019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.676948    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68" event={"ID":"960e55b4-4d31-4210-b756-28c039d88f7f","Type":"ContainerDied","Data":"a9c1e7fbaaa240065fa5eb1c5d6883fc06cb3a2a147bd1939af744de12ec5c51"}
Nov 02 13:34:48.677019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.676977    5879 scope.go:122] "RemoveContainer" containerID="a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983"
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.689931    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-certs\") pod \"960e55b4-4d31-4210-b756-28c039d88f7f\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") "
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.689979    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960e55b4-4d31-4210-b756-28c039d88f7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960e55b4-4d31-4210-b756-28c039d88f7f-istiod-ca-cert\") pod \"960e55b4-4d31-4210-b756-28c039d88f7f\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") "
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.690027    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-credential-socket\") pod \"960e55b4-4d31-4210-b756-28c039d88f7f\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") "
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.690064    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-socket\") pod \"960e55b4-4d31-4210-b756-28c039d88f7f\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") "
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.690106    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-istio-token\" (UniqueName: \"kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-istio-token\") pod \"960e55b4-4d31-4210-b756-28c039d88f7f\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") "
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.690146    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/960e55b4-4d31-4210-b756-28c039d88f7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960e55b4-4d31-4210-b756-28c039d88f7f-istio-podinfo\") pod \"960e55b4-4d31-4210-b756-28c039d88f7f\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") "
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.690238    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-data\") pod \"960e55b4-4d31-4210-b756-28c039d88f7f\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") "
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.690331    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-kube-api-access-dbrkj\" (UniqueName: \"kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-kube-api-access-dbrkj\") pod \"960e55b4-4d31-4210-b756-28c039d88f7f\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") "
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.690371    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-envoy\") pod \"960e55b4-4d31-4210-b756-28c039d88f7f\" (UID: \"960e55b4-4d31-4210-b756-28c039d88f7f\") "
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.691986    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-certs" pod "960e55b4-4d31-4210-b756-28c039d88f7f" (UID: "960e55b4-4d31-4210-b756-28c039d88f7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.692681    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960e55b4-4d31-4210-b756-28c039d88f7f-istiod-ca-cert" pod "960e55b4-4d31-4210-b756-28c039d88f7f" (UID: "960e55b4-4d31-4210-b756-28c039d88f7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.692979    5879 scope.go:122] "RemoveContainer" containerID="a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983"
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.693313    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-data" pod "960e55b4-4d31-4210-b756-28c039d88f7f" (UID: "960e55b4-4d31-4210-b756-28c039d88f7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:34:48.693488    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983\": container with ID starting with a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983 not found: ID does not exist" containerID="a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983"
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.693521    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983"} err="failed to get container status \"a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983\": rpc error: code = NotFound desc = could not find container \"a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983\": container with ID starting with a495d5c104fde7229c38666ca057b8a1eaf6d5dabe187285abe5e026bee11983 not found: ID does not exist"
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.693537    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-credential-socket" pod "960e55b4-4d31-4210-b756-28c039d88f7f" (UID: "960e55b4-4d31-4210-b756-28c039d88f7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.694638    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-socket" pod "960e55b4-4d31-4210-b756-28c039d88f7f" (UID: "960e55b4-4d31-4210-b756-28c039d88f7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:48.697228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.696879    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-envoy" pod "960e55b4-4d31-4210-b756-28c039d88f7f" (UID: "960e55b4-4d31-4210-b756-28c039d88f7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:48.698255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.697420    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-istio-token" pod "960e55b4-4d31-4210-b756-28c039d88f7f" (UID: "960e55b4-4d31-4210-b756-28c039d88f7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:48.698255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.697979    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-kube-api-access-dbrkj" pod "960e55b4-4d31-4210-b756-28c039d88f7f" (UID: "960e55b4-4d31-4210-b756-28c039d88f7f"). InnerVolumeSpecName "kube-api-access-dbrkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:48.698387 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.698349    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/960e55b4-4d31-4210-b756-28c039d88f7f-istio-podinfo" pod "960e55b4-4d31-4210-b756-28c039d88f7f" (UID: "960e55b4-4d31-4210-b756-28c039d88f7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:34:48.802688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.792451    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.802688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.792503    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/960e55b4-4d31-4210-b756-28c039d88f7f-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.802688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.792533    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.802688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.792555    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.802688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.792574    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.802688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.792593    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/960e55b4-4d31-4210-b756-28c039d88f7f-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.802688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.792613    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.802688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.792629    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrkj\" (UniqueName: \"kubernetes.io/projected/960e55b4-4d31-4210-b756-28c039d88f7f-kube-api-access-dbrkj\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.802688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.792645    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/960e55b4-4d31-4210-b756-28c039d88f7f-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:48.863829 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:48.863790    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:49.029930 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:49.029891    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"]
Nov 02 13:34:49.041823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:49.041792    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-61e341-openshift-default-59d97476dc-88q68"]
Nov 02 13:34:49.853549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:49.853507    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main" containerID="cri-o://f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1" gracePeriod=45
Nov 02 13:34:49.857682 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:49.857649    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main" containerID="cri-o://7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee" gracePeriod=45
Nov 02 13:34:49.870429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:49.870399    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main" containerID="cri-o://854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73" gracePeriod=45
Nov 02 13:34:50.023660 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.023613    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" path="/var/lib/kubelet/pods/960e55b4-4d31-4210-b756-28c039d88f7f/volumes"
Nov 02 13:34:50.120405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.120366    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:50.174245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.172532    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 13:34:50.181345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.181317    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 13:34:50.200150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.200122    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 13:34:50.206850 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.206795    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-model-cache\") pod \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") "
Nov 02 13:34:50.207042 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.206917    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-home\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-home\") pod \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") "
Nov 02 13:34:50.207042 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.206957    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-kube-api-access-dltpn\" (UniqueName: \"kubernetes.io/projected/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-kube-api-access-dltpn\") pod \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") "
Nov 02 13:34:50.207166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207044    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-dshm\") pod \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") "
Nov 02 13:34:50.207166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207075    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-dshm\") pod \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") "
Nov 02 13:34:50.207166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207137    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-kube-api-access-jjxlx\" (UniqueName: \"kubernetes.io/projected/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-kube-api-access-jjxlx\") pod \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") "
Nov 02 13:34:50.207166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207139    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-model-cache" pod "6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" (UID: "6db32bc0-dcc9-4edf-9829-caf8a9ec4a70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.207388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207185    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tmp-dir\") pod \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") "
Nov 02 13:34:50.207388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207214    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tls-certs\" (UniqueName: \"kubernetes.io/secret/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tls-certs\") pod \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") "
Nov 02 13:34:50.207388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207239    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tls-certs\" (UniqueName: \"kubernetes.io/secret/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tls-certs\") pod \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") "
Nov 02 13:34:50.207388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207271    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-home\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-home\") pod \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") "
Nov 02 13:34:50.207388 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207317    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tmp-dir\") pod \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\" (UID: \"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70\") "
Nov 02 13:34:50.207720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207428    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-model-cache\") pod \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\" (UID: \"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da\") "
Nov 02 13:34:50.207720 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207265    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-home" pod "6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" (UID: "6db32bc0-dcc9-4edf-9829-caf8a9ec4a70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.208407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207927    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.208407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207953    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.208407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.207500    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tmp-dir" pod "91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" (UID: "91b3cc5f-ed9c-4662-9889-ee99a4a8c0da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.208407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.208026    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-model-cache" pod "91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" (UID: "91b3cc5f-ed9c-4662-9889-ee99a4a8c0da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.208407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.208239    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tmp-dir" pod "6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" (UID: "6db32bc0-dcc9-4edf-9829-caf8a9ec4a70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.208407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.208264    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-home" pod "91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" (UID: "91b3cc5f-ed9c-4662-9889-ee99a4a8c0da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.210236 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.210209    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-dshm" pod "6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" (UID: "6db32bc0-dcc9-4edf-9829-caf8a9ec4a70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.210393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.210370    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tls-certs" pod "6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" (UID: "6db32bc0-dcc9-4edf-9829-caf8a9ec4a70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:50.210480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.210415    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-kube-api-access-jjxlx" pod "91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" (UID: "91b3cc5f-ed9c-4662-9889-ee99a4a8c0da"). InnerVolumeSpecName "kube-api-access-jjxlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:50.210480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.210444    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-kube-api-access-dltpn" pod "6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" (UID: "6db32bc0-dcc9-4edf-9829-caf8a9ec4a70"). InnerVolumeSpecName "kube-api-access-dltpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:50.211202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.211180    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tls-certs" pod "91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" (UID: "91b3cc5f-ed9c-4662-9889-ee99a4a8c0da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:50.211395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.211371    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-dshm" pod "91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" (UID: "91b3cc5f-ed9c-4662-9889-ee99a4a8c0da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.309652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.309615    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-kube-api-access-r8qzm\" (UniqueName: \"kubernetes.io/projected/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-kube-api-access-r8qzm\") pod \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") "
Nov 02 13:34:50.309844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.309690    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-model-cache\") pod \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") "
Nov 02 13:34:50.309844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.309730    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-dshm\") pod \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") "
Nov 02 13:34:50.309844 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.309815    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-home\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-home\") pod \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") "
Nov 02 13:34:50.310119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.309844    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tls-certs\") pod \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") "
Nov 02 13:34:50.310119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.309890    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tmp-dir\") pod \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\" (UID: \"c2514e25-9ef1-4ff7-b352-b3eff9cecb76\") "
Nov 02 13:34:50.310479 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.310450    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dltpn\" (UniqueName: \"kubernetes.io/projected/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-kube-api-access-dltpn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.310567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.310485    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.310567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.310500    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.310567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.310512    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjxlx\" (UniqueName: \"kubernetes.io/projected/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-kube-api-access-jjxlx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.310567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.310524    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.310567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.310537    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.310567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.310549    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.310567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.310561    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.310961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.310572    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.310961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.310583    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.310961 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.310834    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tmp-dir" pod "c2514e25-9ef1-4ff7-b352-b3eff9cecb76" (UID: "c2514e25-9ef1-4ff7-b352-b3eff9cecb76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.311804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.311434    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-home" pod "c2514e25-9ef1-4ff7-b352-b3eff9cecb76" (UID: "c2514e25-9ef1-4ff7-b352-b3eff9cecb76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.311804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.311450    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-model-cache" pod "c2514e25-9ef1-4ff7-b352-b3eff9cecb76" (UID: "c2514e25-9ef1-4ff7-b352-b3eff9cecb76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.316070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.315689    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-dshm" pod "c2514e25-9ef1-4ff7-b352-b3eff9cecb76" (UID: "c2514e25-9ef1-4ff7-b352-b3eff9cecb76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:34:50.316070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.316059    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-kube-api-access-r8qzm" pod "c2514e25-9ef1-4ff7-b352-b3eff9cecb76" (UID: "c2514e25-9ef1-4ff7-b352-b3eff9cecb76"). InnerVolumeSpecName "kube-api-access-r8qzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:34:50.316395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.316362    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tls-certs" pod "c2514e25-9ef1-4ff7-b352-b3eff9cecb76" (UID: "c2514e25-9ef1-4ff7-b352-b3eff9cecb76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:34:50.411882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.411785    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8qzm\" (UniqueName: \"kubernetes.io/projected/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-kube-api-access-r8qzm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.411882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.411826    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.411882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.411841    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.411882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.411853    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.411882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.411865    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.411882 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.411879    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2514e25-9ef1-4ff7-b352-b3eff9cecb76-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:34:50.556973 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.556934    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:50.694828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.694731    5879 generic.go:356] "Generic (PLEG): container finished" podID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerID="7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee" exitCode=0
Nov 02 13:34:50.695033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.694828    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"
Nov 02 13:34:50.695033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.694873    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" event={"ID":"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70","Type":"ContainerDied","Data":"7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee"}
Nov 02 13:34:50.695033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.694920    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj" event={"ID":"6db32bc0-dcc9-4edf-9829-caf8a9ec4a70","Type":"ContainerDied","Data":"2fab048514b6c3d47f354e21cc1ef472dd0a57dad19467c5b34f09697272e4c5"}
Nov 02 13:34:50.695033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.694938    5879 scope.go:122] "RemoveContainer" containerID="7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee"
Nov 02 13:34:50.696719 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.696698    5879 generic.go:356] "Generic (PLEG): container finished" podID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerID="854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73" exitCode=0
Nov 02 13:34:50.696859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.696748    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"
Nov 02 13:34:50.696859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.696780    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" event={"ID":"c2514e25-9ef1-4ff7-b352-b3eff9cecb76","Type":"ContainerDied","Data":"854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73"}
Nov 02 13:34:50.696859 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.696830    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb" event={"ID":"c2514e25-9ef1-4ff7-b352-b3eff9cecb76","Type":"ContainerDied","Data":"9e185482927bddd88497453f1d8e099a49c35316720d1194fa87978b0b67bd79"}
Nov 02 13:34:50.698594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.698569    5879 generic.go:356] "Generic (PLEG): container finished" podID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerID="f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1" exitCode=0
Nov 02 13:34:50.698731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.698607    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"
Nov 02 13:34:50.698731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.698669    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" event={"ID":"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da","Type":"ContainerDied","Data":"f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1"}
Nov 02 13:34:50.698731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.698720    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s" event={"ID":"91b3cc5f-ed9c-4662-9889-ee99a4a8c0da","Type":"ContainerDied","Data":"6b9736b78a743fb981364d7842f862dd2e167b121b98e211428e9f3fd29b462f"}
Nov 02 13:34:50.708153 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.708124    5879 scope.go:122] "RemoveContainer" containerID="7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee"
Nov 02 13:34:50.708466 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:34:50.708448    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee\": container with ID starting with 7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee not found: ID does not exist" containerID="7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee"
Nov 02 13:34:50.708524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.708475    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee"} err="failed to get container status \"7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee\": rpc error: code = NotFound desc = could not find container \"7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee\": container with ID starting with 7b286e0e9f2ed3866f37208c789535ebc59dc5f1cebe993bd0f026df852f8aee not found: ID does not exist"
Nov 02 13:34:50.708524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.708493    5879 scope.go:122] "RemoveContainer" containerID="854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73"
Nov 02 13:34:50.719841 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.719818    5879 scope.go:122] "RemoveContainer" containerID="854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73"
Nov 02 13:34:50.720167 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:34:50.720145    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73\": container with ID starting with 854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73 not found: ID does not exist" containerID="854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73"
Nov 02 13:34:50.720239 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.720179    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73"} err="failed to get container status \"854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73\": rpc error: code = NotFound desc = could not find container \"854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73\": container with ID starting with 854cc6e3d1107b4a7053f17003478d0c2fe0b40ce79c72c6f8bc8dee2180cf73 not found: ID does not exist"
Nov 02 13:34:50.720239 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.720200    5879 scope.go:122] "RemoveContainer" containerID="f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1"
Nov 02 13:34:50.730266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.730247    5879 scope.go:122] "RemoveContainer" containerID="f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1"
Nov 02 13:34:50.730547 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:34:50.730527    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1\": container with ID starting with f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1 not found: ID does not exist" containerID="f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1"
Nov 02 13:34:50.730607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.730558    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1"} err="failed to get container status \"f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1\": rpc error: code = NotFound desc = could not find container \"f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1\": container with ID starting with f346efec22d579af30b0135e63b3e98dbdcd9cdf7effaffc791910b3875886f1 not found: ID does not exist"
Nov 02 13:34:50.751931 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.751895    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"]
Nov 02 13:34:50.765662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.765630    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/distinct-w2-95ee26-kserve-69ff98695f-4947s"]
Nov 02 13:34:50.792046 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.792006    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"]
Nov 02 13:34:50.796262 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.796231    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/simulator-w2-95ee26-kserve-54c446c865-tbvjb"]
Nov 02 13:34:50.833854 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.833818    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"]
Nov 02 13:34:50.837578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:50.837553    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-95ee26/premium-w2-95ee26-kserve-5b9f48d64d-zkbvj"]
Nov 02 13:34:51.031712 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:51.031629    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"]
Nov 02 13:34:51.035015 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:51.034975    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-253c20b6-bspjw"]
Nov 02 13:34:52.018122 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:52.018091    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" path="/var/lib/kubelet/pods/6db32bc0-dcc9-4edf-9829-caf8a9ec4a70/volumes"
Nov 02 13:34:52.018549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:52.018536    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f574761-e825-4e38-a1d6-3bd749376635" path="/var/lib/kubelet/pods/6f574761-e825-4e38-a1d6-3bd749376635/volumes"
Nov 02 13:34:52.018909 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:52.018897    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" path="/var/lib/kubelet/pods/91b3cc5f-ed9c-4662-9889-ee99a4a8c0da/volumes"
Nov 02 13:34:52.019631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:52.019615    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" path="/var/lib/kubelet/pods/c2514e25-9ef1-4ff7-b352-b3eff9cecb76/volumes"
Nov 02 13:34:52.175717 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:52.175686    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:53.840789 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:53.840744    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:55.728468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:55.728427    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:57.350207 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:57.350175    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:57.665783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:57.665751    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:34:59.055659 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:34:59.055624    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:00.144658 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:00.144616    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:00.639430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:00.639395    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:02.040820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.040783    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"]
Nov 02 13:35:02.054349 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.054317    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-385ad4f2-zz8gq"]
Nov 02 13:35:02.297886 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.297787    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:02.306185 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306150    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"]
Nov 02 13:35:02.306693 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306678    5879 cpu_manager.go:414] "containerMap: removing container" podUID="72323342-920f-4310-8cb8-9a0c558b285a" containerName="revoke-keys"
Nov 02 13:35:02.306744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306695    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72323342-920f-4310-8cb8-9a0c558b285a" containerName="revoke-keys"
Nov 02 13:35:02.306744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306704    5879 cpu_manager.go:414] "containerMap: removing container" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy"
Nov 02 13:35:02.306744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306710    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy"
Nov 02 13:35:02.306744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306729    5879 cpu_manager.go:414] "containerMap: removing container" podUID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" containerName="istio-proxy"
Nov 02 13:35:02.306744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306734    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" containerName="istio-proxy"
Nov 02 13:35:02.306744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306742    5879 cpu_manager.go:414] "containerMap: removing container" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306748    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306757    5879 cpu_manager.go:414] "containerMap: removing container" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306762    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306769    5879 cpu_manager.go:414] "containerMap: removing container" podUID="0c33d267-6b2a-41e0-ba1c-685284c7137e" containerName="payload-pre-processing"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306774    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c33d267-6b2a-41e0-ba1c-685284c7137e" containerName="payload-pre-processing"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306804    5879 cpu_manager.go:414] "containerMap: removing container" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306809    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306818    5879 cpu_manager.go:414] "containerMap: removing container" podUID="63f632d1-51bb-4f53-b7eb-afc0dff0cce2" containerName="payload-processing"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306823    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f632d1-51bb-4f53-b7eb-afc0dff0cce2" containerName="payload-processing"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306897    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2514e25-9ef1-4ff7-b352-b3eff9cecb76" containerName="main"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306906    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b3cc5f-ed9c-4662-9889-ee99a4a8c0da" containerName="main"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306912    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c33d267-6b2a-41e0-ba1c-685284c7137e" containerName="payload-pre-processing"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306923    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db32bc0-dcc9-4edf-9829-caf8a9ec4a70" containerName="main"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306928    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f632d1-51bb-4f53-b7eb-afc0dff0cce2" containerName="payload-processing"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306937    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="72323342-920f-4310-8cb8-9a0c558b285a" containerName="revoke-keys"
Nov 02 13:35:02.306940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306945    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="960e55b4-4d31-4210-b756-28c039d88f7f" containerName="istio-proxy"
Nov 02 13:35:02.307489 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.306952    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cfe2d10-a2c9-4145-abe6-3a9e66b1e9cc" containerName="istio-proxy"
Nov 02 13:35:02.312392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.312372    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:02.329256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.329218    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:35:02.330067 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.330036    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w9cwp\""
Nov 02 13:35:02.333120 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.333089    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"]
Nov 02 13:35:02.448518 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.448484    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d5a1784-6whqq\" (UID: \"30d52089-5ede-45b2-90b8-c2e4fc32359b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:02.448711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.448646    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgpsk\" (UniqueName: \"kubernetes.io/projected/30d52089-5ede-45b2-90b8-c2e4fc32359b-kube-api-access-xgpsk\") pod \"maas-api-revoke-sub-0d5a1784-6whqq\" (UID: \"30d52089-5ede-45b2-90b8-c2e4fc32359b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:02.448711 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.448706    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d5a1784-6whqq\" (UID: \"30d52089-5ede-45b2-90b8-c2e4fc32359b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:02.550483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.550380    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d5a1784-6whqq\" (UID: \"30d52089-5ede-45b2-90b8-c2e4fc32359b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:02.550483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.550463    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d5a1784-6whqq\" (UID: \"30d52089-5ede-45b2-90b8-c2e4fc32359b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:02.550752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.550555    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgpsk\" (UniqueName: \"kubernetes.io/projected/30d52089-5ede-45b2-90b8-c2e4fc32359b-kube-api-access-xgpsk\") pod \"maas-api-revoke-sub-0d5a1784-6whqq\" (UID: \"30d52089-5ede-45b2-90b8-c2e4fc32359b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:02.551282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.551254    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d5a1784-6whqq\" (UID: \"30d52089-5ede-45b2-90b8-c2e4fc32359b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:02.553053 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.553019    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d5a1784-6whqq\" (UID: \"30d52089-5ede-45b2-90b8-c2e4fc32359b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:02.576582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.576549    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgpsk\" (UniqueName: \"kubernetes.io/projected/30d52089-5ede-45b2-90b8-c2e4fc32359b-kube-api-access-xgpsk\") pod \"maas-api-revoke-sub-0d5a1784-6whqq\" (UID: \"30d52089-5ede-45b2-90b8-c2e4fc32359b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:02.627102 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.627065    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:02.928896 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:35:02.928862    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d52089_5ede_45b2_90b8_c2e4fc32359b.slice/crio-532430a553487ae26e0a2e5767e4dc833bb648bd44053515c445233a958cf061 WatchSource:0}: Error finding container 532430a553487ae26e0a2e5767e4dc833bb648bd44053515c445233a958cf061: Status 404 returned error can't find the container with id 532430a553487ae26e0a2e5767e4dc833bb648bd44053515c445233a958cf061
Nov 02 13:35:02.929586 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:02.929557    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"]
Nov 02 13:35:03.790295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:03.790260    5879 generic.go:356] "Generic (PLEG): container finished" podID="30d52089-5ede-45b2-90b8-c2e4fc32359b" containerID="4ceebc075672ef47d62c57b1bdb2e0b65ce38675bfa90008039903c67e4db893" exitCode=0
Nov 02 13:35:03.790730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:03.790346    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq" event={"ID":"30d52089-5ede-45b2-90b8-c2e4fc32359b","Type":"ContainerDied","Data":"4ceebc075672ef47d62c57b1bdb2e0b65ce38675bfa90008039903c67e4db893"}
Nov 02 13:35:03.790730 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:03.790403    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq" event={"ID":"30d52089-5ede-45b2-90b8-c2e4fc32359b","Type":"ContainerStarted","Data":"532430a553487ae26e0a2e5767e4dc833bb648bd44053515c445233a958cf061"}
Nov 02 13:35:03.957874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:03.957771    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:04.017777 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:04.017738    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048eb183-2cd4-49fc-b6fc-2c4662a29637" path="/var/lib/kubelet/pods/048eb183-2cd4-49fc-b6fc-2c4662a29637/volumes"
Nov 02 13:35:04.949817 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:04.949788    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:05.078889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.078799    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-xks-ca\") pod \"30d52089-5ede-45b2-90b8-c2e4fc32359b\" (UID: \"30d52089-5ede-45b2-90b8-c2e4fc32359b\") "
Nov 02 13:35:05.078889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.078870    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-openshift-ca\") pod \"30d52089-5ede-45b2-90b8-c2e4fc32359b\" (UID: \"30d52089-5ede-45b2-90b8-c2e4fc32359b\") "
Nov 02 13:35:05.079201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.078910    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d52089-5ede-45b2-90b8-c2e4fc32359b-kube-api-access-xgpsk\" (UniqueName: \"kubernetes.io/projected/30d52089-5ede-45b2-90b8-c2e4fc32359b-kube-api-access-xgpsk\") pod \"30d52089-5ede-45b2-90b8-c2e4fc32359b\" (UID: \"30d52089-5ede-45b2-90b8-c2e4fc32359b\") "
Nov 02 13:35:05.079309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.079284    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-openshift-ca" pod "30d52089-5ede-45b2-90b8-c2e4fc32359b" (UID: "30d52089-5ede-45b2-90b8-c2e4fc32359b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:35:05.081288 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.081258    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-xks-ca" pod "30d52089-5ede-45b2-90b8-c2e4fc32359b" (UID: "30d52089-5ede-45b2-90b8-c2e4fc32359b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:35:05.081424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.081336    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d52089-5ede-45b2-90b8-c2e4fc32359b-kube-api-access-xgpsk" pod "30d52089-5ede-45b2-90b8-c2e4fc32359b" (UID: "30d52089-5ede-45b2-90b8-c2e4fc32359b"). InnerVolumeSpecName "kube-api-access-xgpsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:35:05.180349 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.180307    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:35:05.180349 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.180350    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d52089-5ede-45b2-90b8-c2e4fc32359b-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:35:05.180349 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.180367    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgpsk\" (UniqueName: \"kubernetes.io/projected/30d52089-5ede-45b2-90b8-c2e4fc32359b-kube-api-access-xgpsk\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:35:05.624763 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.624728    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:05.803302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.803273    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"
Nov 02 13:35:05.803483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.803313    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq" event={"ID":"30d52089-5ede-45b2-90b8-c2e4fc32359b","Type":"ContainerDied","Data":"532430a553487ae26e0a2e5767e4dc833bb648bd44053515c445233a958cf061"}
Nov 02 13:35:05.803483 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:05.803360    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="532430a553487ae26e0a2e5767e4dc833bb648bd44053515c445233a958cf061"
Nov 02 13:35:07.017644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:07.017601    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:07.647190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:07.647160    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:09.584632 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:09.584597    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:10.737311 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:10.737275    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:10.984755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:10.984717    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:11.251279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:11.251253    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:12.929360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:12.929324    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:13.057847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:13.057806    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:14.081668 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:14.081592    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:14.650073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:14.649966    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:14.821906 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:35:14.821873    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:35:15.091279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:15.091190    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:15.293240 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:15.293208    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:15.810049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:15.809971    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:16.212547 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:16.212520    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:16.693096 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:16.693053    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:18.417987 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:18.417954    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:19.982201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:19.982161    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:20.116302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:20.116269    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:21.779229 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:21.779188    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:23.355391 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:23.355317    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:24.941949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:24.941891    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:26.577202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:26.577170    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:28.157617 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:28.157584    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:29.740956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:29.740917    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:31.324134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:31.324088    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:32.877057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:32.877019    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:34.537002 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:34.536967    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:36.158384 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:36.158350    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:36.603871 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:36.603757    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:36.739225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:36.739193    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:37.859335 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:37.859303    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:39.644457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:39.644424    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:41.232081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:41.232043    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:41.941430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:41.941401    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:42.858338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:42.858297    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:44.173328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:44.173299    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:44.442268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:44.442174    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:44.832315 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:35:44.832210    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:35:44.892326 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:44.892291    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:46.050547 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:46.050514    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:47.663601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:47.663551    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:49.296585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:49.296551    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:50.933160 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:50.933130    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:52.507678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:52.507638    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:53.021929 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:53.021890    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:53.448648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:53.448614    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:54.066308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:54.066275    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:55.033544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:55.033504    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"]
Nov 02 13:35:55.047244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:55.047202    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a6b4f0-lh5rh"]
Nov 02 13:35:55.782137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:55.782095    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:56.017313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:56.017273    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341e7f6b-9cb5-4546-b050-9e2bb8f6d130" path="/var/lib/kubelet/pods/341e7f6b-9cb5-4546-b050-9e2bb8f6d130/volumes"
Nov 02 13:35:57.424598 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:57.424565    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:58.967331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:58.967263    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:59.483625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:59.483588    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"]
Nov 02 13:35:59.643796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:59.643757    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"]
Nov 02 13:35:59.776770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:59.776686    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:35:59.906377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:35:59.906331    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:00.717949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:00.717919    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:00.943394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:00.943340    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:02.547744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:02.547706    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:03.977122 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:03.977081    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:04.484780 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:04.484705    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n" podUID="e974ec08-ae6a-490c-946d-f04c6e9ea509" containerName="payload-processing" containerID="cri-o://2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb" gracePeriod=25
Nov 02 13:36:04.608206 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:04.608167    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:04.644843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:04.644773    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg" podUID="ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659" containerName="payload-pre-processing" containerID="cri-o://08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e" gracePeriod=25
Nov 02 13:36:06.030488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:06.030450    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"]
Nov 02 13:36:06.044405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:06.044376    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-662453b6-tbsw7"]
Nov 02 13:36:06.247370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:06.247325    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:06.752260 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:06.752226    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:06.887158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:06.887123    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:07.505649 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:07.505611    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:07.907127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:07.907086    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:08.015653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:08.015618    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aecfd315-f66b-41c1-902f-c10db91c7b96" path="/var/lib/kubelet/pods/aecfd315-f66b-41c1-902f-c10db91c7b96/volumes"
Nov 02 13:36:08.045389 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:08.045346    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:08.682369 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:08.682332    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:09.434916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:09.434867    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n" podUID="e974ec08-ae6a-490c-946d-f04c6e9ea509" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:36:09.434916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:09.434889    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg" podUID="ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:36:09.759228 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:09.759154    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:09.888688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:09.888647    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:11.460060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:11.460021    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:13.165582 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:13.165519    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:14.738004 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:14.737962    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"]
Nov 02 13:36:14.764745 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:14.764706    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:14.776755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:14.776718    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"]
Nov 02 13:36:14.822295 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:36:14.822257    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:36:14.924723 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:14.924681    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:15.060236 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:15.060133    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:15.514190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:15.514145    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:16.464688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:16.464633    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:18.254752 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:18.254717    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:19.434247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:19.434208    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg" podUID="ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:36:19.434247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:19.434229    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n" podUID="e974ec08-ae6a-490c-946d-f04c6e9ea509" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:36:19.738772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:19.738672    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk" podUID="1ca730eb-ca79-4abe-8598-02769a71b915" containerName="payload-processing" containerID="cri-o://2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004" gracePeriod=25
Nov 02 13:36:19.777756 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:19.777689    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc" podUID="5d845a4a-e70b-4b49-b1ea-f3bd4224439b" containerName="payload-pre-processing" containerID="cri-o://27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270" gracePeriod=25
Nov 02 13:36:19.891669 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:19.891622    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:21.162510 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:21.162474    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc" podUID="5d845a4a-e70b-4b49-b1ea-f3bd4224439b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:36:21.203623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:21.203585    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:21.481224 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:21.481109    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:21.756543 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:21.756456    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:22.173257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:22.173221    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk" podUID="1ca730eb-ca79-4abe-8598-02769a71b915" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:36:23.348564 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:23.348533    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:24.774784 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:24.774754    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"
Nov 02 13:36:24.893205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:24.893180    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"
Nov 02 13:36:24.897186 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:24.897165    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e974ec08-ae6a-490c-946d-f04c6e9ea509-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e974ec08-ae6a-490c-946d-f04c6e9ea509-plugins-config-volume\") pod \"e974ec08-ae6a-490c-946d-f04c6e9ea509\" (UID: \"e974ec08-ae6a-490c-946d-f04c6e9ea509\") "
Nov 02 13:36:24.897277 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:24.897225    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e974ec08-ae6a-490c-946d-f04c6e9ea509-kube-api-access-vpkzq\" (UniqueName: \"kubernetes.io/projected/e974ec08-ae6a-490c-946d-f04c6e9ea509-kube-api-access-vpkzq\") pod \"e974ec08-ae6a-490c-946d-f04c6e9ea509\" (UID: \"e974ec08-ae6a-490c-946d-f04c6e9ea509\") "
Nov 02 13:36:24.897521 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:24.897501    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e974ec08-ae6a-490c-946d-f04c6e9ea509-plugins-config-volume" pod "e974ec08-ae6a-490c-946d-f04c6e9ea509" (UID: "e974ec08-ae6a-490c-946d-f04c6e9ea509"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:24.899833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:24.899803    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e974ec08-ae6a-490c-946d-f04c6e9ea509-kube-api-access-vpkzq" pod "e974ec08-ae6a-490c-946d-f04c6e9ea509" (UID: "e974ec08-ae6a-490c-946d-f04c6e9ea509"). InnerVolumeSpecName "kube-api-access-vpkzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:24.998073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:24.998042    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-plugins-config-volume\") pod \"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659\" (UID: \"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659\") "
Nov 02 13:36:24.998268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:24.998089    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-kube-api-access-wxvcc\" (UniqueName: \"kubernetes.io/projected/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-kube-api-access-wxvcc\") pod \"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659\" (UID: \"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659\") "
Nov 02 13:36:24.998432 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:24.998420    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e974ec08-ae6a-490c-946d-f04c6e9ea509-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:24.998471 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:24.998434    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpkzq\" (UniqueName: \"kubernetes.io/projected/e974ec08-ae6a-490c-946d-f04c6e9ea509-kube-api-access-vpkzq\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:24.998525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:24.998499    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-plugins-config-volume" pod "ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659" (UID: "ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:25.000528 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.000479    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-kube-api-access-wxvcc" pod "ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659" (UID: "ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659"). InnerVolumeSpecName "kube-api-access-wxvcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:25.070857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.070821    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:25.099539 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.099498    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:25.099539 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.099532    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxvcc\" (UniqueName: \"kubernetes.io/projected/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659-kube-api-access-wxvcc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:25.314150 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.314049    5879 generic.go:356] "Generic (PLEG): container finished" podID="ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659" containerID="08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e" exitCode=0
Nov 02 13:36:25.314339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.314208    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg" event={"ID":"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659","Type":"ContainerDied","Data":"08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e"}
Nov 02 13:36:25.314407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.314363    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg" event={"ID":"ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659","Type":"ContainerDied","Data":"5f61f491c9d84be17b714715aa932bebb5772e6e27f6e68e9b983b678a71f90d"}
Nov 02 13:36:25.314407 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.314401    5879 scope.go:122] "RemoveContainer" containerID="08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e"
Nov 02 13:36:25.314987 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.314615    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"
Nov 02 13:36:25.316678 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.316649    5879 generic.go:356] "Generic (PLEG): container finished" podID="e974ec08-ae6a-490c-946d-f04c6e9ea509" containerID="2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb" exitCode=0
Nov 02 13:36:25.316840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.316759    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"
Nov 02 13:36:25.319366 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.316778    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n" event={"ID":"e974ec08-ae6a-490c-946d-f04c6e9ea509","Type":"ContainerDied","Data":"2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb"}
Nov 02 13:36:25.319512 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.319411    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n" event={"ID":"e974ec08-ae6a-490c-946d-f04c6e9ea509","Type":"ContainerDied","Data":"e0460e4b82766c8dd858ab46d88511b6f526aee641147b00fe24f7a27e04f1d9"}
Nov 02 13:36:25.330401 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.330379    5879 scope.go:122] "RemoveContainer" containerID="08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e"
Nov 02 13:36:25.330726 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:36:25.330706    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e\": container with ID starting with 08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e not found: ID does not exist" containerID="08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e"
Nov 02 13:36:25.330782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.330737    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e"} err="failed to get container status \"08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e\": rpc error: code = NotFound desc = could not find container \"08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e\": container with ID starting with 08aa0af290994f087915ff045b0828db311d096495368e578554a5d505f0235e not found: ID does not exist"
Nov 02 13:36:25.330782 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.330754    5879 scope.go:122] "RemoveContainer" containerID="2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb"
Nov 02 13:36:25.340527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.340498    5879 scope.go:122] "RemoveContainer" containerID="2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb"
Nov 02 13:36:25.340848 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:36:25.340826    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb\": container with ID starting with 2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb not found: ID does not exist" containerID="2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb"
Nov 02 13:36:25.340925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.340858    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb"} err="failed to get container status \"2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb\": rpc error: code = NotFound desc = could not find container \"2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb\": container with ID starting with 2f3ffa047f92db163aa975074cf27ad13d466cb0a23cf4d9490fa44f34b32aeb not found: ID does not exist"
Nov 02 13:36:25.373102 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.373065    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"]
Nov 02 13:36:25.386950 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.386913    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-14fcb6-677f77795f-p496n"]
Nov 02 13:36:25.424295 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.424255    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"]
Nov 02 13:36:25.427875 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:25.427850    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-14fcb6-86d4dd6867-cf6jg"]
Nov 02 13:36:26.017096 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:26.017059    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e974ec08-ae6a-490c-946d-f04c6e9ea509" path="/var/lib/kubelet/pods/e974ec08-ae6a-490c-946d-f04c6e9ea509/volumes"
Nov 02 13:36:26.017524 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:26.017467    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659" path="/var/lib/kubelet/pods/ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659/volumes"
Nov 02 13:36:26.672430 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:26.672401    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:27.908059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:27.907931    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:28.032712 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:28.032673    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:28.415085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:28.415047    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:28.699803 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:28.699703    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:29.030915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:29.030813    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"]
Nov 02 13:36:29.031491 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:29.031102    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy" containerID="cri-o://a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f" gracePeriod=30
Nov 02 13:36:29.082035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:29.081231    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:29.746511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:29.746480    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:29.881106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:29.881061    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:30.252288 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:30.252252    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:31.162544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:31.162502    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc" podUID="5d845a4a-e70b-4b49-b1ea-f3bd4224439b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:36:31.866761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:31.866722    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:32.173224 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:32.173136    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk" podUID="1ca730eb-ca79-4abe-8598-02769a71b915" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:36:33.466718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:33.466683    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:34.288714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.288691    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:36:34.375723 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.375689    5879 generic.go:356] "Generic (PLEG): container finished" podID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerID="a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f" exitCode=0
Nov 02 13:36:34.375925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.375745    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"
Nov 02 13:36:34.375925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.375778    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" event={"ID":"40d9f2c8-011b-49fa-9f7b-5a36ee074541","Type":"ContainerDied","Data":"a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f"}
Nov 02 13:36:34.375925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.375859    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx" event={"ID":"40d9f2c8-011b-49fa-9f7b-5a36ee074541","Type":"ContainerDied","Data":"3da61e0e02850ef6630e752ffa8bec0ed6edfbfd3d4f92aeb3bdb0c0ff2b8d06"}
Nov 02 13:36:34.375925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.375895    5879 scope.go:122] "RemoveContainer" containerID="a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f"
Nov 02 13:36:34.386685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.386660    5879 scope.go:122] "RemoveContainer" containerID="f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd"
Nov 02 13:36:34.396925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.396905    5879 scope.go:122] "RemoveContainer" containerID="a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f"
Nov 02 13:36:34.397335 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:36:34.397312    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f\": container with ID starting with a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f not found: ID does not exist" containerID="a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f"
Nov 02 13:36:34.397403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.397349    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f"} err="failed to get container status \"a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f\": rpc error: code = NotFound desc = could not find container \"a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f\": container with ID starting with a6b74734835b25954e618b71989156f1fb8fd88d83cbe271c6cdda79a452081f not found: ID does not exist"
Nov 02 13:36:34.397403 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.397375    5879 scope.go:122] "RemoveContainer" containerID="f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd"
Nov 02 13:36:34.397630 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:36:34.397610    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd\": container with ID starting with f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd not found: ID does not exist" containerID="f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd"
Nov 02 13:36:34.397680 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.397637    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd"} err="failed to get container status \"f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd\": rpc error: code = NotFound desc = could not find container \"f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd\": container with ID starting with f88b3fb313037e4d52f4d1fc7efd9a3d73feaa4dba17bd00dfba5709b71650bd not found: ID does not exist"
Nov 02 13:36:34.401966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.401942    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-podinfo\") pod \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") "
Nov 02 13:36:34.402121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.402090    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-envoy\") pod \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") "
Nov 02 13:36:34.402192 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.402152    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istiod-ca-cert\") pod \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") "
Nov 02 13:36:34.402251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.402214    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-kube-api-access-txbrq\" (UniqueName: \"kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-kube-api-access-txbrq\") pod \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") "
Nov 02 13:36:34.402371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.402348    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-socket\") pod \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") "
Nov 02 13:36:34.402441 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.402425    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-token\" (UniqueName: \"kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-token\") pod \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") "
Nov 02 13:36:34.402501 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.402469    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-certs\") pod \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") "
Nov 02 13:36:34.402558 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.402546    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-data\") pod \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") "
Nov 02 13:36:34.402615 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.402594    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-socket" pod "40d9f2c8-011b-49fa-9f7b-5a36ee074541" (UID: "40d9f2c8-011b-49fa-9f7b-5a36ee074541"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:34.402653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.402612    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-credential-socket\") pod \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\" (UID: \"40d9f2c8-011b-49fa-9f7b-5a36ee074541\") "
Nov 02 13:36:34.402820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.402797    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-certs" pod "40d9f2c8-011b-49fa-9f7b-5a36ee074541" (UID: "40d9f2c8-011b-49fa-9f7b-5a36ee074541"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:34.402903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.402817    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-data" pod "40d9f2c8-011b-49fa-9f7b-5a36ee074541" (UID: "40d9f2c8-011b-49fa-9f7b-5a36ee074541"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:34.402903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.402860    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istiod-ca-cert" pod "40d9f2c8-011b-49fa-9f7b-5a36ee074541" (UID: "40d9f2c8-011b-49fa-9f7b-5a36ee074541"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:34.403118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.403004    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-credential-socket" pod "40d9f2c8-011b-49fa-9f7b-5a36ee074541" (UID: "40d9f2c8-011b-49fa-9f7b-5a36ee074541"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:34.403360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.403339    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:34.403429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.403366    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:34.403429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.403381    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:34.403429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.403398    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:34.403429 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.403413    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:34.404755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.404733    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-envoy" pod "40d9f2c8-011b-49fa-9f7b-5a36ee074541" (UID: "40d9f2c8-011b-49fa-9f7b-5a36ee074541"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:36:34.404853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.404768    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-token" pod "40d9f2c8-011b-49fa-9f7b-5a36ee074541" (UID: "40d9f2c8-011b-49fa-9f7b-5a36ee074541"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:34.404908 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.404861    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-kube-api-access-txbrq" pod "40d9f2c8-011b-49fa-9f7b-5a36ee074541" (UID: "40d9f2c8-011b-49fa-9f7b-5a36ee074541"). InnerVolumeSpecName "kube-api-access-txbrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:34.404908 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.404879    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-podinfo" pod "40d9f2c8-011b-49fa-9f7b-5a36ee074541" (UID: "40d9f2c8-011b-49fa-9f7b-5a36ee074541"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:36:34.504476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.504438    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:34.504476 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.504472    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:34.504919 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.504486    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txbrq\" (UniqueName: \"kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-kube-api-access-txbrq\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:34.504919 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.504502    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40d9f2c8-011b-49fa-9f7b-5a36ee074541-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:34.725639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.725589    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"]
Nov 02 13:36:34.748839 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:34.748806    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e3396f-openshift-default-5c76f8c6d5-jk8bx"]
Nov 02 13:36:35.479021 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:35.478949    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:36.017095 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:36.017052    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" path="/var/lib/kubelet/pods/40d9f2c8-011b-49fa-9f7b-5a36ee074541/volumes"
Nov 02 13:36:36.385135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:36.385087    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:36.541348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:36.541310    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:37.128956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:37.128924    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:37.262747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:37.262708    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:38.473836 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:38.473804    5879 scope.go:122] "RemoveContainer" containerID="98298fb6ee52ee5bd3d9edc978641dbad6719577fac29df34f5decd7d1a4c8bb"
Nov 02 13:36:38.483204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:38.483182    5879 scope.go:122] "RemoveContainer" containerID="b384542edea209b2fb0319c990e5b54f206fb91f8ccf9f46a439a2b3e92b0e4f"
Nov 02 13:36:38.757653 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:38.757568    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:40.037578 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.037551    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"
Nov 02 13:36:40.072249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.072219    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"
Nov 02 13:36:40.167594 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.167561    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca730eb-ca79-4abe-8598-02769a71b915-kube-api-access-q5c4v\" (UniqueName: \"kubernetes.io/projected/1ca730eb-ca79-4abe-8598-02769a71b915-kube-api-access-q5c4v\") pod \"1ca730eb-ca79-4abe-8598-02769a71b915\" (UID: \"1ca730eb-ca79-4abe-8598-02769a71b915\") "
Nov 02 13:36:40.167804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.167683    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca730eb-ca79-4abe-8598-02769a71b915-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca730eb-ca79-4abe-8598-02769a71b915-plugins-config-volume\") pod \"1ca730eb-ca79-4abe-8598-02769a71b915\" (UID: \"1ca730eb-ca79-4abe-8598-02769a71b915\") "
Nov 02 13:36:40.168085 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.168062    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca730eb-ca79-4abe-8598-02769a71b915-plugins-config-volume" pod "1ca730eb-ca79-4abe-8598-02769a71b915" (UID: "1ca730eb-ca79-4abe-8598-02769a71b915"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:40.169928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.169906    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca730eb-ca79-4abe-8598-02769a71b915-kube-api-access-q5c4v" pod "1ca730eb-ca79-4abe-8598-02769a71b915" (UID: "1ca730eb-ca79-4abe-8598-02769a71b915"). InnerVolumeSpecName "kube-api-access-q5c4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:40.269266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.269170    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-plugins-config-volume\") pod \"5d845a4a-e70b-4b49-b1ea-f3bd4224439b\" (UID: \"5d845a4a-e70b-4b49-b1ea-f3bd4224439b\") "
Nov 02 13:36:40.269266 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.269240    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-kube-api-access-dw6wc\" (UniqueName: \"kubernetes.io/projected/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-kube-api-access-dw6wc\") pod \"5d845a4a-e70b-4b49-b1ea-f3bd4224439b\" (UID: \"5d845a4a-e70b-4b49-b1ea-f3bd4224439b\") "
Nov 02 13:36:40.269576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.269552    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-plugins-config-volume" pod "5d845a4a-e70b-4b49-b1ea-f3bd4224439b" (UID: "5d845a4a-e70b-4b49-b1ea-f3bd4224439b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:36:40.269645 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.269620    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5c4v\" (UniqueName: \"kubernetes.io/projected/1ca730eb-ca79-4abe-8598-02769a71b915-kube-api-access-q5c4v\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:40.269699 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.269645    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca730eb-ca79-4abe-8598-02769a71b915-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:40.271641 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.271622    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-kube-api-access-dw6wc" pod "5d845a4a-e70b-4b49-b1ea-f3bd4224439b" (UID: "5d845a4a-e70b-4b49-b1ea-f3bd4224439b"). InnerVolumeSpecName "kube-api-access-dw6wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:36:40.370315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.370275    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:40.370315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.370309    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw6wc\" (UniqueName: \"kubernetes.io/projected/5d845a4a-e70b-4b49-b1ea-f3bd4224439b-kube-api-access-dw6wc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:36:40.407479 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.407439    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:40.422665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.422634    5879 generic.go:356] "Generic (PLEG): container finished" podID="1ca730eb-ca79-4abe-8598-02769a71b915" containerID="2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004" exitCode=0
Nov 02 13:36:40.422845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.422679    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"
Nov 02 13:36:40.422845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.422717    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk" event={"ID":"1ca730eb-ca79-4abe-8598-02769a71b915","Type":"ContainerDied","Data":"2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004"}
Nov 02 13:36:40.422845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.422772    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk" event={"ID":"1ca730eb-ca79-4abe-8598-02769a71b915","Type":"ContainerDied","Data":"d25ec05ecba41a6e0a0e785ef42f648a981b5a9081b8e5850f673edfbc87c2fc"}
Nov 02 13:36:40.422845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.422797    5879 scope.go:122] "RemoveContainer" containerID="2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004"
Nov 02 13:36:40.424117 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.424095    5879 generic.go:356] "Generic (PLEG): container finished" podID="5d845a4a-e70b-4b49-b1ea-f3bd4224439b" containerID="27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270" exitCode=0
Nov 02 13:36:40.424254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.424126    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc" event={"ID":"5d845a4a-e70b-4b49-b1ea-f3bd4224439b","Type":"ContainerDied","Data":"27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270"}
Nov 02 13:36:40.424254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.424148    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"
Nov 02 13:36:40.424254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.424174    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc" event={"ID":"5d845a4a-e70b-4b49-b1ea-f3bd4224439b","Type":"ContainerDied","Data":"a2945ad6edafaeb010c2e8b973d00d2ad88126f6d0b53c264bf870b4a6cb6fe5"}
Nov 02 13:36:40.435571 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.435545    5879 scope.go:122] "RemoveContainer" containerID="2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004"
Nov 02 13:36:40.435913 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:36:40.435887    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004\": container with ID starting with 2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004 not found: ID does not exist" containerID="2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004"
Nov 02 13:36:40.435977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.435926    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004"} err="failed to get container status \"2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004\": rpc error: code = NotFound desc = could not find container \"2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004\": container with ID starting with 2b76fbb413af75c6888f128462bfb5845fedaf90a39a17462d1f67ab0759a004 not found: ID does not exist"
Nov 02 13:36:40.435977 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.435950    5879 scope.go:122] "RemoveContainer" containerID="27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270"
Nov 02 13:36:40.445737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.445719    5879 scope.go:122] "RemoveContainer" containerID="27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270"
Nov 02 13:36:40.446053 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:36:40.446036    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270\": container with ID starting with 27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270 not found: ID does not exist" containerID="27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270"
Nov 02 13:36:40.446105 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.446064    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270"} err="failed to get container status \"27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270\": rpc error: code = NotFound desc = could not find container \"27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270\": container with ID starting with 27512744aade8e42e486c0c9a3cba8f6950d331addaab28acec7ffdb9be13270 not found: ID does not exist"
Nov 02 13:36:40.479778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.479744    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"]
Nov 02 13:36:40.486154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.486122    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e3396f-5b4fcb8f5-gjxrc"]
Nov 02 13:36:40.523134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.523059    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"]
Nov 02 13:36:40.526843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:40.526819    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e3396f-6b84fd8ccc-9f6dk"]
Nov 02 13:36:41.992971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:41.992939    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:42.015480 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:42.015450    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca730eb-ca79-4abe-8598-02769a71b915" path="/var/lib/kubelet/pods/1ca730eb-ca79-4abe-8598-02769a71b915/volumes"
Nov 02 13:36:42.015840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:42.015827    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d845a4a-e70b-4b49-b1ea-f3bd4224439b" path="/var/lib/kubelet/pods/5d845a4a-e70b-4b49-b1ea-f3bd4224439b/volumes"
Nov 02 13:36:42.114864 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:42.114827    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:43.228619 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:43.228579    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:43.369040 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:43.368983    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:43.550273 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:43.550137    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:43.975108 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:43.975071    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:44.408845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:44.408810    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:44.821610 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:36:44.821524    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:36:45.182919 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:45.182883    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:45.625297 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:45.625258    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:47.261793 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:47.261758    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:48.846680 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:48.846650    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:49.678275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:49.678223    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:49.815246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:49.815212    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:50.464325 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:50.464270    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:50.727621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:50.727547    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"]
Nov 02 13:36:50.761083 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:50.761045    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"]
Nov 02 13:36:50.903956 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:50.903923    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:51.035636 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:51.035542    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:51.887175 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:51.887139    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:52.384662 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:52.384615    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:52.522210 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:52.522164    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:54.066770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:54.066717    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:55.569734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:55.569693    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:55.728767 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:55.728688    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh" podUID="b99cd497-48ae-4d1d-adc8-4c4b7ab018f3" containerName="payload-processing" containerID="cri-o://fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c" gracePeriod=25
Nov 02 13:36:55.761529 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:55.761471    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf" podUID="47674527-cd9d-480e-b292-776c382c168d" containerName="payload-pre-processing" containerID="cri-o://869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1" gracePeriod=25
Nov 02 13:36:57.134368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:57.134327    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:57.929175 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:57.929135    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:58.068304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:58.068274    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:58.720776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:58.720743    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:58.856843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:58.856808    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:59.040176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:59.040089    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"]
Nov 02 13:36:59.054199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:59.054170    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c8f7ecc-47r48"]
Nov 02 13:36:59.509559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:59.509526    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:36:59.852329 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:36:59.852235    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:00.016321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:00.016287    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af88dd3b-20a3-4706-8fa7-103c53c5b66e" path="/var/lib/kubelet/pods/af88dd3b-20a3-4706-8fa7-103c53c5b66e/volumes"
Nov 02 13:37:00.031216 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:00.031184    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"]
Nov 02 13:37:00.045303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:00.045273    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"]
Nov 02 13:37:00.047204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:00.047188    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3f703db-srdlf"]
Nov 02 13:37:00.048883 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:00.048862    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d12afe-wc4rg"]
Nov 02 13:37:00.328030 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:00.327956    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:00.647567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:00.647534    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:02.015231 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:02.015200    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46900508-af5c-462d-9cd5-c5ef358bca6d" path="/var/lib/kubelet/pods/46900508-af5c-462d-9cd5-c5ef358bca6d/volumes"
Nov 02 13:37:02.015636 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:02.015623    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2" path="/var/lib/kubelet/pods/dd0fc01d-9af0-4ea1-9563-66d78a5e7fe2/volumes"
Nov 02 13:37:02.372964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:02.372927    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:04.057741 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:04.057711    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:05.010903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:05.010866    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh" podUID="b99cd497-48ae-4d1d-adc8-4c4b7ab018f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:37:05.010903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:05.010889    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf" podUID="47674527-cd9d-480e-b292-776c382c168d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:37:05.251949 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:05.251908    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"]
Nov 02 13:37:05.267753 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:05.267664    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"]
Nov 02 13:37:05.408012 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:05.407956    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:05.655761 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:05.655725    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:06.170303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:06.170265    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:07.266374 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:07.266338    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:08.945526 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:08.945483    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:09.242466 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:09.242376    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:09.382963 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:09.382925    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:10.031657 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:10.031626    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"]
Nov 02 13:37:10.045731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:10.045693    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df8d5c7-qvjb2"]
Nov 02 13:37:10.047555 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:10.047531    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"]
Nov 02 13:37:10.049208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:10.049174    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"]
Nov 02 13:37:10.050800 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:10.050768    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9743aa45-ddmg5"]
Nov 02 13:37:10.059133 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:10.059107    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a30862bf-vw8v4"]
Nov 02 13:37:10.547009 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:10.546959    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:12.016209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:12.016173    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6260358-d500-464b-bf83-b400a5f5d497" path="/var/lib/kubelet/pods/a6260358-d500-464b-bf83-b400a5f5d497/volumes"
Nov 02 13:37:12.016652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:12.016643    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b39179e3-ef04-4cab-b41e-51e0990a2c84" path="/var/lib/kubelet/pods/b39179e3-ef04-4cab-b41e-51e0990a2c84/volumes"
Nov 02 13:37:12.017058 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:12.017041    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d29d64-f04e-432f-8136-0315436fad62" path="/var/lib/kubelet/pods/b5d29d64-f04e-432f-8136-0315436fad62/volumes"
Nov 02 13:37:12.177118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:12.177058    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:12.648592 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:12.648553    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:12.796447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:12.796412    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:13.454396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:13.454368    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:14.819792 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:37:14.819760    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:37:15.010808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:15.010771    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf" podUID="47674527-cd9d-480e-b292-776c382c168d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:37:15.011037 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:15.010775    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh" podUID="b99cd497-48ae-4d1d-adc8-4c4b7ab018f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 13:37:16.028861 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.028833    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"
Nov 02 13:37:16.057648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.057619    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"
Nov 02 13:37:16.067356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.067333    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-kube-api-access-dbjxn\" (UniqueName: \"kubernetes.io/projected/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-kube-api-access-dbjxn\") pod \"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3\" (UID: \"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3\") "
Nov 02 13:37:16.067487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.067475    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-plugins-config-volume\") pod \"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3\" (UID: \"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3\") "
Nov 02 13:37:16.067820 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.067798    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-plugins-config-volume" pod "b99cd497-48ae-4d1d-adc8-4c4b7ab018f3" (UID: "b99cd497-48ae-4d1d-adc8-4c4b7ab018f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:16.069666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.069640    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-kube-api-access-dbjxn" pod "b99cd497-48ae-4d1d-adc8-4c4b7ab018f3" (UID: "b99cd497-48ae-4d1d-adc8-4c4b7ab018f3"). InnerVolumeSpecName "kube-api-access-dbjxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:16.168236 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.168196    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47674527-cd9d-480e-b292-776c382c168d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47674527-cd9d-480e-b292-776c382c168d-plugins-config-volume\") pod \"47674527-cd9d-480e-b292-776c382c168d\" (UID: \"47674527-cd9d-480e-b292-776c382c168d\") "
Nov 02 13:37:16.168442 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.168293    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47674527-cd9d-480e-b292-776c382c168d-kube-api-access-lpr25\" (UniqueName: \"kubernetes.io/projected/47674527-cd9d-480e-b292-776c382c168d-kube-api-access-lpr25\") pod \"47674527-cd9d-480e-b292-776c382c168d\" (UID: \"47674527-cd9d-480e-b292-776c382c168d\") "
Nov 02 13:37:16.168691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.168665    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47674527-cd9d-480e-b292-776c382c168d-plugins-config-volume" pod "47674527-cd9d-480e-b292-776c382c168d" (UID: "47674527-cd9d-480e-b292-776c382c168d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:16.168842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.168825    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47674527-cd9d-480e-b292-776c382c168d-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:16.168921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.168845    5879 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-plugins-config-volume\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:16.168921 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.168854    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbjxn\" (UniqueName: \"kubernetes.io/projected/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3-kube-api-access-dbjxn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:16.170762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.170733    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47674527-cd9d-480e-b292-776c382c168d-kube-api-access-lpr25" pod "47674527-cd9d-480e-b292-776c382c168d" (UID: "47674527-cd9d-480e-b292-776c382c168d"). InnerVolumeSpecName "kube-api-access-lpr25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:16.269685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.269598    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpr25\" (UniqueName: \"kubernetes.io/projected/47674527-cd9d-480e-b292-776c382c168d-kube-api-access-lpr25\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:16.646057 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.646014    5879 generic.go:356] "Generic (PLEG): container finished" podID="47674527-cd9d-480e-b292-776c382c168d" containerID="869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1" exitCode=0
Nov 02 13:37:16.646272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.646078    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"
Nov 02 13:37:16.646272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.646115    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf" event={"ID":"47674527-cd9d-480e-b292-776c382c168d","Type":"ContainerDied","Data":"869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1"}
Nov 02 13:37:16.646272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.646173    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf" event={"ID":"47674527-cd9d-480e-b292-776c382c168d","Type":"ContainerDied","Data":"979cd7561738a7b6d565467320e5da0c0c0bcb2507ae188e41d1c505033e8f31"}
Nov 02 13:37:16.646272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.646201    5879 scope.go:122] "RemoveContainer" containerID="869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1"
Nov 02 13:37:16.647802 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.647778    5879 generic.go:356] "Generic (PLEG): container finished" podID="b99cd497-48ae-4d1d-adc8-4c4b7ab018f3" containerID="fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c" exitCode=0
Nov 02 13:37:16.647951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.647835    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"
Nov 02 13:37:16.647951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.647884    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh" event={"ID":"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3","Type":"ContainerDied","Data":"fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c"}
Nov 02 13:37:16.647951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.647938    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh" event={"ID":"b99cd497-48ae-4d1d-adc8-4c4b7ab018f3","Type":"ContainerDied","Data":"ec3a852fafe0e4e39b4671a4d44127eb068e0cbd2baf52be7dc748b6f570aa0a"}
Nov 02 13:37:16.657059 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.656916    5879 scope.go:122] "RemoveContainer" containerID="869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1"
Nov 02 13:37:16.657328 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:37:16.657303    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1\": container with ID starting with 869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1 not found: ID does not exist" containerID="869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1"
Nov 02 13:37:16.657449 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.657335    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1"} err="failed to get container status \"869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1\": rpc error: code = NotFound desc = could not find container \"869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1\": container with ID starting with 869dd635b27050d6a67a1311a001d87094afbaaf7ef53c4734bdb174e3ead7b1 not found: ID does not exist"
Nov 02 13:37:16.657449 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.657353    5879 scope.go:122] "RemoveContainer" containerID="fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c"
Nov 02 13:37:16.666574 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.666555    5879 scope.go:122] "RemoveContainer" containerID="fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c"
Nov 02 13:37:16.666854 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:37:16.666835    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c\": container with ID starting with fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c not found: ID does not exist" containerID="fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c"
Nov 02 13:37:16.666903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.666862    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c"} err="failed to get container status \"fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c\": rpc error: code = NotFound desc = could not find container \"fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c\": container with ID starting with fd4417d9e8e71d6fc1f6bc8eac4bf0ba8a5b8f85ae4dcc14707d99fc753ea95c not found: ID does not exist"
Nov 02 13:37:16.702750 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.702717    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"]
Nov 02 13:37:16.716251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.716223    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-43be82-5566f49cff-4jpxh"]
Nov 02 13:37:16.749434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.749409    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"]
Nov 02 13:37:16.761462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:16.761434    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-43be82-649f8bfdff-8w6jf"]
Nov 02 13:37:18.015351 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:18.015319    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47674527-cd9d-480e-b292-776c382c168d" path="/var/lib/kubelet/pods/47674527-cd9d-480e-b292-776c382c168d/volumes"
Nov 02 13:37:18.015943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:18.015677    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99cd497-48ae-4d1d-adc8-4c4b7ab018f3" path="/var/lib/kubelet/pods/b99cd497-48ae-4d1d-adc8-4c4b7ab018f3/volumes"
Nov 02 13:37:19.667215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:19.667178    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:20.276287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.276250    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" containerID="cri-o://2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370" gracePeriod=45
Nov 02 13:37:20.293448 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.293416    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" containerID="cri-o://a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d" gracePeriod=45
Nov 02 13:37:20.372112 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.372084    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused"
Nov 02 13:37:20.575190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.575168    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 13:37:20.578916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.578897    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 13:37:20.612012 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.611968    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-tmp-dir\") pod \"171a46b7-75bc-4212-8cb7-3252b2001291\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") "
Nov 02 13:37:20.612148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612054    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-home\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-home\") pod \"171a46b7-75bc-4212-8cb7-3252b2001291\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") "
Nov 02 13:37:20.612148 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612084    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-model-cache\") pod \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") "
Nov 02 13:37:20.612482 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612395    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/171a46b7-75bc-4212-8cb7-3252b2001291-tls-certs\" (UniqueName: \"kubernetes.io/secret/171a46b7-75bc-4212-8cb7-3252b2001291-tls-certs\") pod \"171a46b7-75bc-4212-8cb7-3252b2001291\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") "
Nov 02 13:37:20.612579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612375    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-tmp-dir" pod "171a46b7-75bc-4212-8cb7-3252b2001291" (UID: "171a46b7-75bc-4212-8cb7-3252b2001291"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.612579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612503    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-dshm\") pod \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") "
Nov 02 13:37:20.612579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612558    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/171a46b7-75bc-4212-8cb7-3252b2001291-kube-api-access-9fqtl\" (UniqueName: \"kubernetes.io/projected/171a46b7-75bc-4212-8cb7-3252b2001291-kube-api-access-9fqtl\") pod \"171a46b7-75bc-4212-8cb7-3252b2001291\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") "
Nov 02 13:37:20.612783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612578    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-home" pod "171a46b7-75bc-4212-8cb7-3252b2001291" (UID: "171a46b7-75bc-4212-8cb7-3252b2001291"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.612783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612593    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tmp-dir\") pod \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") "
Nov 02 13:37:20.612783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612645    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-home\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-home\") pod \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") "
Nov 02 13:37:20.612783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612690    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-dshm\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-dshm\") pod \"171a46b7-75bc-4212-8cb7-3252b2001291\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") "
Nov 02 13:37:20.612783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612702    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-model-cache" pod "4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" (UID: "4f25ccf0-beac-4f85-b0ac-b08d8a9591ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.612783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612729    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-kube-api-access-4tfhq\" (UniqueName: \"kubernetes.io/projected/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-kube-api-access-4tfhq\") pod \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") "
Nov 02 13:37:20.612783 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612765    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tls-certs\") pod \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\" (UID: \"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab\") "
Nov 02 13:37:20.613225 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.612914    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tmp-dir" pod "4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" (UID: "4f25ccf0-beac-4f85-b0ac-b08d8a9591ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.614395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.613715    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-home" pod "4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" (UID: "4f25ccf0-beac-4f85-b0ac-b08d8a9591ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.614395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.613863    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-model-cache\") pod \"171a46b7-75bc-4212-8cb7-3252b2001291\" (UID: \"171a46b7-75bc-4212-8cb7-3252b2001291\") "
Nov 02 13:37:20.614395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.614346    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-model-cache" pod "171a46b7-75bc-4212-8cb7-3252b2001291" (UID: "171a46b7-75bc-4212-8cb7-3252b2001291"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.614395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.614354    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.614395 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.614394    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.614727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.614405    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.614727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.614413    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.614727 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.614421    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.615595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.615571    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-dshm" pod "4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" (UID: "4f25ccf0-beac-4f85-b0ac-b08d8a9591ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.616164 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.616127    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tls-certs" pod "4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" (UID: "4f25ccf0-beac-4f85-b0ac-b08d8a9591ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:37:20.617099 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.617078    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-kube-api-access-4tfhq" pod "4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" (UID: "4f25ccf0-beac-4f85-b0ac-b08d8a9591ab"). InnerVolumeSpecName "kube-api-access-4tfhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:20.617919 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.617903    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/171a46b7-75bc-4212-8cb7-3252b2001291-tls-certs" pod "171a46b7-75bc-4212-8cb7-3252b2001291" (UID: "171a46b7-75bc-4212-8cb7-3252b2001291"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:37:20.618194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.618177    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171a46b7-75bc-4212-8cb7-3252b2001291-kube-api-access-9fqtl" pod "171a46b7-75bc-4212-8cb7-3252b2001291" (UID: "171a46b7-75bc-4212-8cb7-3252b2001291"). InnerVolumeSpecName "kube-api-access-9fqtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:20.618289 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.618270    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-dshm" pod "171a46b7-75bc-4212-8cb7-3252b2001291" (UID: "171a46b7-75bc-4212-8cb7-3252b2001291"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:20.692605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.692565    5879 generic.go:356] "Generic (PLEG): container finished" podID="171a46b7-75bc-4212-8cb7-3252b2001291" containerID="a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d" exitCode=0
Nov 02 13:37:20.692605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.692615    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"
Nov 02 13:37:20.693215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.692654    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" event={"ID":"171a46b7-75bc-4212-8cb7-3252b2001291","Type":"ContainerDied","Data":"a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d"}
Nov 02 13:37:20.693215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.692695    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" event={"ID":"171a46b7-75bc-4212-8cb7-3252b2001291","Type":"ContainerDied","Data":"3c24b08047dbaf87edbbeaa36418e6660b2025268efe7c840ab9c2140c1cd93f"}
Nov 02 13:37:20.693215 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.692716    5879 scope.go:122] "RemoveContainer" containerID="a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d"
Nov 02 13:37:20.695423 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.695396    5879 generic.go:356] "Generic (PLEG): container finished" podID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerID="2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370" exitCode=0
Nov 02 13:37:20.695561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.695449    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"
Nov 02 13:37:20.695561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.695505    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" event={"ID":"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab","Type":"ContainerDied","Data":"2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370"}
Nov 02 13:37:20.695561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.695552    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc" event={"ID":"4f25ccf0-beac-4f85-b0ac-b08d8a9591ab","Type":"ContainerDied","Data":"b01809c224065bb22d0990e885aca5694d9e7295aa54eada9a3f6628664d6b05"}
Nov 02 13:37:20.704876 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.704551    5879 scope.go:122] "RemoveContainer" containerID="a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d"
Nov 02 13:37:20.704876 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:37:20.704855    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d\": container with ID starting with a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d not found: ID does not exist" containerID="a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d"
Nov 02 13:37:20.705082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.704886    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d"} err="failed to get container status \"a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d\": rpc error: code = NotFound desc = could not find container \"a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d\": container with ID starting with a1085dfc379ae2457ccac5ef3890b9e4247d37f1d5d9011adaaa078c63f0ac4d not found: ID does not exist"
Nov 02 13:37:20.705082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.704929    5879 scope.go:122] "RemoveContainer" containerID="2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370"
Nov 02 13:37:20.715253 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.715233    5879 scope.go:122] "RemoveContainer" containerID="2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370"
Nov 02 13:37:20.715393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.715286    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/171a46b7-75bc-4212-8cb7-3252b2001291-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.715393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.715305    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.715393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.715314    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fqtl\" (UniqueName: \"kubernetes.io/projected/171a46b7-75bc-4212-8cb7-3252b2001291-kube-api-access-9fqtl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.715393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.715324    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.715393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.715333    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tfhq\" (UniqueName: \"kubernetes.io/projected/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-kube-api-access-4tfhq\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.715393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.715343    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.715393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.715355    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/171a46b7-75bc-4212-8cb7-3252b2001291-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:20.715818 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:37:20.715689    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370\": container with ID starting with 2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370 not found: ID does not exist" containerID="2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370"
Nov 02 13:37:20.715818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.715708    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370"} err="failed to get container status \"2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370\": rpc error: code = NotFound desc = could not find container \"2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370\": container with ID starting with 2dd4ed4e65bf1cdc8c0c41c8dd5388f39e38d4ab52d593eafb5c11c45f92c370 not found: ID does not exist"
Nov 02 13:37:20.748847 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.748823    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"]
Nov 02 13:37:20.755375 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.755352    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv"]
Nov 02 13:37:20.792567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.792544    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"]
Nov 02 13:37:20.796734 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:20.796667    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5514e5/distinct-2-w0-5514e5-kserve-6fdf65ff75-8t5rc"]
Nov 02 13:37:21.040724 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:21.040693    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"]
Nov 02 13:37:21.042773 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:21.042748    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa8d36f-z5x5k"]
Nov 02 13:37:21.576294 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:21.576212    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-5514e5/premium-w0-5514e5-kserve-785c9dd85c-fqkgv" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:37:22.015864 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:22.015831    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" path="/var/lib/kubelet/pods/171a46b7-75bc-4212-8cb7-3252b2001291/volumes"
Nov 02 13:37:22.016301 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:22.016262    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" path="/var/lib/kubelet/pods/4f25ccf0-beac-4f85-b0ac-b08d8a9591ab/volumes"
Nov 02 13:37:22.016614 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:22.016601    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb97089-bcb3-4101-ae19-2b932d18344c" path="/var/lib/kubelet/pods/6bb97089-bcb3-4101-ae19-2b932d18344c/volumes"
Nov 02 13:37:23.663023 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:23.662969    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:25.610677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:25.610651    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:25.745979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:25.745946    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:31.030808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:31.030775    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"]
Nov 02 13:37:31.044422 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:31.044398    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb70a73-7wrtq"]
Nov 02 13:37:32.015625 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:32.015594    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bebb2ec-9564-4e70-9e56-443a8c8f9a8a" path="/var/lib/kubelet/pods/8bebb2ec-9564-4e70-9e56-443a8c8f9a8a/volumes"
Nov 02 13:37:38.524718 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:38.524683    5879 scope.go:122] "RemoveContainer" containerID="003fc0080fa4d6f6adc4494165b0259d20b8749c373f680ab91c738303c43620"
Nov 02 13:37:38.533797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:38.533774    5879 scope.go:122] "RemoveContainer" containerID="66b0bf5c4fbe8906c6aeb91ea2a88c52633fac45e237a7da1f7f8c0d8d4f0e4f"
Nov 02 13:37:42.196112 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:42.196079    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:42.694184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:42.694154    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:42.979173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:42.979100    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:43.213972 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:43.213944    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:43.400047 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:43.399979    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:44.097922 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:44.097879    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:44.820859 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:37:44.820824    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:37:45.818915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:45.818877    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:46.126804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:46.126764    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"]
Nov 02 13:37:46.127370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:46.127048    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" podUID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" containerName="istio-proxy" containerID="cri-o://1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c" gracePeriod=30
Nov 02 13:37:46.191258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:46.191211    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:47.956156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:47.956126    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rdc2r_d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb/kuadrant-console-plugin/0.log"
Nov 02 13:37:49.905195 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:49.905160    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:49.934204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:49.934177    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7js7d_2f09ae13-0192-4a68-a935-39846f02caca/manager/0.log"
Nov 02 13:37:50.054386 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:50.054357    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:50.125125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:50.125102    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-klw92_0b2da242-fe89-4999-a164-2f251fb97b3f/manager/0.log"
Nov 02 13:37:50.332175 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:50.332108    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vd4d6_8e9e3acc-7a16-4166-bf7d-22cc70517482/manager/0.log"
Nov 02 13:37:50.809431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:50.809403    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5597f6b9f6-blz6m_e2d5d9f7-939e-4c5e-8785-921c13b4ebf4/manager/1.log"
Nov 02 13:37:50.994313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:50.994284    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nwfcr_8891b09c-a283-422f-a4ea-25f84ea846a8/server/0.log"
Nov 02 13:37:51.186915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.186894    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vwd9x_8b097954-45fc-45e8-937e-4ece9fbdc1c6/manager/0.log"
Nov 02 13:37:51.381369 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.381334    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 13:37:51.414322 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.414249    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-podinfo\") pod \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") "
Nov 02 13:37:51.414449 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.414342    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-kube-api-access-22fkw\" (UniqueName: \"kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-kube-api-access-22fkw\") pod \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") "
Nov 02 13:37:51.414449 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.414376    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istiod-ca-cert\") pod \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") "
Nov 02 13:37:51.414449 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.414426    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-credential-socket\") pod \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") "
Nov 02 13:37:51.414655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.414480    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-socket\") pod \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") "
Nov 02 13:37:51.414655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.414524    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-envoy\") pod \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") "
Nov 02 13:37:51.414655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.414551    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-token\" (UniqueName: \"kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-token\") pod \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") "
Nov 02 13:37:51.414655 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.414638    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-data\") pod \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") "
Nov 02 13:37:51.414901 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.414726    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-credential-socket" pod "fc5c93ad-afcd-4b72-b3ab-a04045b4746c" (UID: "fc5c93ad-afcd-4b72-b3ab-a04045b4746c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:51.414901 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.414734    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-certs\") pod \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\" (UID: \"fc5c93ad-afcd-4b72-b3ab-a04045b4746c\") "
Nov 02 13:37:51.415056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.414959    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-certs" pod "fc5c93ad-afcd-4b72-b3ab-a04045b4746c" (UID: "fc5c93ad-afcd-4b72-b3ab-a04045b4746c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:51.415278 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.415258    5879 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-credential-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:51.415371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.415281    5879 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:51.415514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.415495    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-data" pod "fc5c93ad-afcd-4b72-b3ab-a04045b4746c" (UID: "fc5c93ad-afcd-4b72-b3ab-a04045b4746c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:51.415590 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.415517    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-socket" pod "fc5c93ad-afcd-4b72-b3ab-a04045b4746c" (UID: "fc5c93ad-afcd-4b72-b3ab-a04045b4746c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:51.415943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.415862    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istiod-ca-cert" pod "fc5c93ad-afcd-4b72-b3ab-a04045b4746c" (UID: "fc5c93ad-afcd-4b72-b3ab-a04045b4746c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:37:51.416969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.416936    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-podinfo" pod "fc5c93ad-afcd-4b72-b3ab-a04045b4746c" (UID: "fc5c93ad-afcd-4b72-b3ab-a04045b4746c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 13:37:51.417766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.417719    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-envoy" pod "fc5c93ad-afcd-4b72-b3ab-a04045b4746c" (UID: "fc5c93ad-afcd-4b72-b3ab-a04045b4746c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:37:51.417957 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.417921    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-kube-api-access-22fkw" pod "fc5c93ad-afcd-4b72-b3ab-a04045b4746c" (UID: "fc5c93ad-afcd-4b72-b3ab-a04045b4746c"). InnerVolumeSpecName "kube-api-access-22fkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:51.418795 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.418766    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-token" pod "fc5c93ad-afcd-4b72-b3ab-a04045b4746c" (UID: "fc5c93ad-afcd-4b72-b3ab-a04045b4746c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:37:51.420539 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.420519    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cvvr4_637730bd-76f5-4bff-8dc2-ba86e4e5254d/proxy/0.log"
Nov 02 13:37:51.516254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.516225    5879 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-podinfo\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:51.516254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.516253    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22fkw\" (UniqueName: \"kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-kube-api-access-22fkw\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:51.516443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.516263    5879 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istiod-ca-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:51.516443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.516272    5879 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-workload-socket\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:51.516443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.516282    5879 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-envoy\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:51.516443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.516290    5879 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-token\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:51.516443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.516297    5879 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5c93ad-afcd-4b72-b3ab-a04045b4746c-istio-data\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:37:51.868126 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.868091    5879 generic.go:356] "Generic (PLEG): container finished" podID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" containerID="1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c" exitCode=0
Nov 02 13:37:51.868299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.868155    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"
Nov 02 13:37:51.868299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.868191    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" event={"ID":"fc5c93ad-afcd-4b72-b3ab-a04045b4746c","Type":"ContainerDied","Data":"1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c"}
Nov 02 13:37:51.868299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.868266    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw" event={"ID":"fc5c93ad-afcd-4b72-b3ab-a04045b4746c","Type":"ContainerDied","Data":"1edbb8867b1a99514d283e9538b39ef5c3a5749cf5f8a3a873ca3de5a5fa0b70"}
Nov 02 13:37:51.868299 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.868288    5879 scope.go:122] "RemoveContainer" containerID="1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c"
Nov 02 13:37:51.878274 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.878261    5879 scope.go:122] "RemoveContainer" containerID="1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c"
Nov 02 13:37:51.878550 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:37:51.878529    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c\": container with ID starting with 1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c not found: ID does not exist" containerID="1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c"
Nov 02 13:37:51.878605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.878559    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c"} err="failed to get container status \"1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c\": rpc error: code = NotFound desc = could not find container \"1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c\": container with ID starting with 1831df563f20d875c5825580d74cad48b2d446632aa5171e14f39f15e07bf89c not found: ID does not exist"
Nov 02 13:37:51.926452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.926419    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"]
Nov 02 13:37:51.940544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.940519    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-5514e5-openshift-default-7c9c979d5f-6g4sw"]
Nov 02 13:37:51.992605 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:51.992581    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5597f6b9f6-blz6m_e2d5d9f7-939e-4c5e-8785-921c13b4ebf4/manager/0.log"
Nov 02 13:37:52.015626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:52.015602    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" path="/var/lib/kubelet/pods/fc5c93ad-afcd-4b72-b3ab-a04045b4746c/volumes"
Nov 02 13:37:56.220770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:56.220740    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:56.677421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:56.677385    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:56.852414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:56.852374    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:59.331121 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:59.331082    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:37:59.541573 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:37:59.541538    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:00.803202 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:00.803163    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:00.952357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:00.952329    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:01.095136 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:01.095051    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:01.223688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:01.223652    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:02.469117 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:02.469081    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:04.064712 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:04.064682    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:05.710708 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:05.710677    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:07.267777 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:07.267743    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:08.777760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:08.777730    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:10.472435 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:10.472397    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:12.116307 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:12.116278    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:13.651180 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:13.651103    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:14.825124 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:38:14.825086    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:38:15.257770 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:15.257739    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:17.082737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:17.082708    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:18.675828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:18.675794    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:20.288094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:20.288063    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:21.848406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:21.848373    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:23.398758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:23.398715    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:24.971270 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:24.971234    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:25.892854 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:25.892807    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:26.019631 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:26.019597    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:26.358342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:26.358251    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:28.536063 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:28.536032    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:28.677788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:28.677758    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:31.639857 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:31.639818    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:35.507563 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:35.507528    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-blz6m"]
Nov 02 13:38:35.508062 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:35.507739    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5597f6b9f6-blz6m" podUID="e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" containerName="manager" containerID="cri-o://e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5" gracePeriod=30
Nov 02 13:38:35.787466 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:35.787435    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 13:38:35.856393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:35.856337    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-webhook-cert\") pod \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\" (UID: \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\") "
Nov 02 13:38:35.856627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:35.856513    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-kube-api-access-wnbdp\" (UniqueName: \"kubernetes.io/projected/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-kube-api-access-wnbdp\") pod \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\" (UID: \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\") "
Nov 02 13:38:35.856721 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:35.856675    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-metrics-certs\") pod \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\" (UID: \"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4\") "
Nov 02 13:38:35.859633 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:35.859596    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-webhook-cert" pod "e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" (UID: "e2d5d9f7-939e-4c5e-8785-921c13b4ebf4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:38:35.859810 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:35.859764    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-metrics-certs" pod "e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" (UID: "e2d5d9f7-939e-4c5e-8785-921c13b4ebf4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:38:35.859888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:35.859854    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-kube-api-access-wnbdp" pod "e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" (UID: "e2d5d9f7-939e-4c5e-8785-921c13b4ebf4"). InnerVolumeSpecName "kube-api-access-wnbdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:38:35.958691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:35.958633    5879 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-webhook-cert\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:38:35.958691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:35.958677    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnbdp\" (UniqueName: \"kubernetes.io/projected/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-kube-api-access-wnbdp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:38:35.958691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:35.958688    5879 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4-metrics-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:38:36.124445 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:36.124408    5879 generic.go:356] "Generic (PLEG): container finished" podID="e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" containerID="e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5" exitCode=0
Nov 02 13:38:36.124445 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:36.124454    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5597f6b9f6-blz6m" event={"ID":"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4","Type":"ContainerDied","Data":"e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5"}
Nov 02 13:38:36.124757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:36.124500    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5597f6b9f6-blz6m"
Nov 02 13:38:36.124757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:36.124533    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5597f6b9f6-blz6m" event={"ID":"e2d5d9f7-939e-4c5e-8785-921c13b4ebf4","Type":"ContainerDied","Data":"22aa186183c8b54870dd1fce9ae4e7d3c48951b7469880dac06b17672a67aa30"}
Nov 02 13:38:36.124757 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:36.124556    5879 scope.go:122] "RemoveContainer" containerID="e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5"
Nov 02 13:38:36.137618 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:36.137593    5879 scope.go:122] "RemoveContainer" containerID="b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025"
Nov 02 13:38:36.149589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:36.149496    5879 scope.go:122] "RemoveContainer" containerID="e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5"
Nov 02 13:38:36.149797 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:38:36.149775    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5\": container with ID starting with e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5 not found: ID does not exist" containerID="e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5"
Nov 02 13:38:36.149889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:36.149813    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5"} err="failed to get container status \"e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5\": rpc error: code = NotFound desc = could not find container \"e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5\": container with ID starting with e107d8075f8df5e6eef7cbffe96f6f9f82e7927e436926d9decbee5d72ea0fd5 not found: ID does not exist"
Nov 02 13:38:36.149889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:36.149841    5879 scope.go:122] "RemoveContainer" containerID="b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025"
Nov 02 13:38:36.150134 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:38:36.150070    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025\": container with ID starting with b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025 not found: ID does not exist" containerID="b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025"
Nov 02 13:38:36.150134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:36.150110    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025"} err="failed to get container status \"b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025\": rpc error: code = NotFound desc = could not find container \"b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025\": container with ID starting with b8d9c9fd5d85639d14b2d19695a021634fb579bbada269d7e174bf1c454ab025 not found: ID does not exist"
Nov 02 13:38:36.185356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:36.185314    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-blz6m"]
Nov 02 13:38:36.198396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:36.198366    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5597f6b9f6-blz6m"]
Nov 02 13:38:38.015845 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:38.015807    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" path="/var/lib/kubelet/pods/e2d5d9f7-939e-4c5e-8785-921c13b4ebf4/volumes"
Nov 02 13:38:38.587885 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:38.587853    5879 scope.go:122] "RemoveContainer" containerID="562add3f1640e7c523c206eeacd4e394fb1ee648d42f9edf8114c8d3b174e9b3"
Nov 02 13:38:38.597410 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:38.597381    5879 scope.go:122] "RemoveContainer" containerID="b4ed7d9710553938d45744afb1c9122c90fa2d2d7f4df9827906afe7b6b84c73"
Nov 02 13:38:38.606203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:38.606182    5879 scope.go:122] "RemoveContainer" containerID="cbbb43c08b33388daf166f92fca362eddfd6762e15a7b80f1601e116697df33d"
Nov 02 13:38:38.614762 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:38.614738    5879 scope.go:122] "RemoveContainer" containerID="6eca7e38263eb037624f990aa916c20d8c636491a625fc57514dae309b46ad08"
Nov 02 13:38:38.623264 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:38.623240    5879 scope.go:122] "RemoveContainer" containerID="e0c5f0fd55bb55cbcee3f1d6482c4362e887be4734f7f7bee4f3b0ddac18ed5c"
Nov 02 13:38:38.632951 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:38.632927    5879 scope.go:122] "RemoveContainer" containerID="c02dcb828d77c2bea4188723b482e34cc16a42b8ae7786e36b6924adf1e068c7"
Nov 02 13:38:38.641576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:38.641553    5879 scope.go:122] "RemoveContainer" containerID="96fc4cef62ae1d4cf200b99b85d18c4856d5d65dbf84aac7d309ded6c7dfefd9"
Nov 02 13:38:38.650070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:38.650039    5879 scope.go:122] "RemoveContainer" containerID="22155dab2a468501318b068c37aa8296dd58dbe2e2f30cd9112be13dd070cbe6"
Nov 02 13:38:42.548156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:38:42.548081    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:38:44.820532 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:38:44.820495    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:39:01.041925 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:01.041891    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"]
Nov 02 13:39:01.055827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:01.055792    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d37229e2-7594z"]
Nov 02 13:39:02.015654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:02.015615    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dabe41c-5395-4d96-bf7c-c5dc43b5ed37" path="/var/lib/kubelet/pods/8dabe41c-5395-4d96-bf7c-c5dc43b5ed37/volumes"
Nov 02 13:39:12.030514 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:12.030477    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"]
Nov 02 13:39:12.044339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:12.044305    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f965e06-qkfmq"]
Nov 02 13:39:12.419368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:12.419322    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:39:14.015157 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:14.015118    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72323342-920f-4310-8cb8-9a0c558b285a" path="/var/lib/kubelet/pods/72323342-920f-4310-8cb8-9a0c558b285a/volumes"
Nov 02 13:39:14.177552 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:14.177522    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:39:14.821282 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:39:14.821243    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:39:16.197840 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.197796    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"]
Nov 02 13:39:16.198303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198283    5879 cpu_manager.go:414] "containerMap: removing container" podUID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" containerName="istio-proxy"
Nov 02 13:39:16.198303 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198297    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" containerName="istio-proxy"
Nov 02 13:39:16.198380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198315    5879 cpu_manager.go:414] "containerMap: removing container" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy"
Nov 02 13:39:16.198380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198321    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy"
Nov 02 13:39:16.198380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198338    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" containerName="manager"
Nov 02 13:39:16.198380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198345    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" containerName="manager"
Nov 02 13:39:16.198380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198353    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5d845a4a-e70b-4b49-b1ea-f3bd4224439b" containerName="payload-pre-processing"
Nov 02 13:39:16.198380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198358    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d845a4a-e70b-4b49-b1ea-f3bd4224439b" containerName="payload-pre-processing"
Nov 02 13:39:16.198380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198369    5879 cpu_manager.go:414] "containerMap: removing container" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main"
Nov 02 13:39:16.198380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198374    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main"
Nov 02 13:39:16.198380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198381    5879 cpu_manager.go:414] "containerMap: removing container" podUID="30d52089-5ede-45b2-90b8-c2e4fc32359b" containerName="revoke-keys"
Nov 02 13:39:16.198380 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198386    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d52089-5ede-45b2-90b8-c2e4fc32359b" containerName="revoke-keys"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198396    5879 cpu_manager.go:414] "containerMap: removing container" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198400    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198411    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e974ec08-ae6a-490c-946d-f04c6e9ea509" containerName="payload-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198419    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e974ec08-ae6a-490c-946d-f04c6e9ea509" containerName="payload-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198432    5879 cpu_manager.go:414] "containerMap: removing container" podUID="47674527-cd9d-480e-b292-776c382c168d" containerName="payload-pre-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198437    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47674527-cd9d-480e-b292-776c382c168d" containerName="payload-pre-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198443    5879 cpu_manager.go:414] "containerMap: removing container" podUID="b99cd497-48ae-4d1d-adc8-4c4b7ab018f3" containerName="payload-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198447    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99cd497-48ae-4d1d-adc8-4c4b7ab018f3" containerName="payload-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198454    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" containerName="manager"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198458    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" containerName="manager"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198463    5879 cpu_manager.go:414] "containerMap: removing container" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198467    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198479    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659" containerName="payload-pre-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198484    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659" containerName="payload-pre-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198498    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1ca730eb-ca79-4abe-8598-02769a71b915" containerName="payload-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198505    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca730eb-ca79-4abe-8598-02769a71b915" containerName="payload-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198563    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198573    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" containerName="manager"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198585    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d845a4a-e70b-4b49-b1ea-f3bd4224439b" containerName="payload-pre-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198591    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f25ccf0-beac-4f85-b0ac-b08d8a9591ab" containerName="main"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198596    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d52089-5ede-45b2-90b8-c2e4fc32359b" containerName="revoke-keys"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198601    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="171a46b7-75bc-4212-8cb7-3252b2001291" containerName="main"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198607    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e974ec08-ae6a-490c-946d-f04c6e9ea509" containerName="payload-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198613    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="47674527-cd9d-480e-b292-776c382c168d" containerName="payload-pre-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198618    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99cd497-48ae-4d1d-adc8-4c4b7ab018f3" containerName="payload-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198623    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d5d9f7-939e-4c5e-8785-921c13b4ebf4" containerName="manager"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198629    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee045bd7-66e5-4bd3-97c9-1c0e1e7c8659" containerName="payload-pre-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198634    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca730eb-ca79-4abe-8598-02769a71b915" containerName="payload-processing"
Nov 02 13:39:16.198706 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.198641    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5c93ad-afcd-4b72-b3ab-a04045b4746c" containerName="istio-proxy"
Nov 02 13:39:16.202194 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.202169    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.234924 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.234890    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"]
Nov 02 13:39:16.284259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.284219    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.284259 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.284269    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.284515 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.284290    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.284515 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.284384    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.284587 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.284509    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.284627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.284600    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.284665 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.284625    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.284707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.284699    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfg76\" (UniqueName: \"kubernetes.io/projected/3a8de801-f2f2-4b08-ad24-022a2e063bc8-kube-api-access-qfg76\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.284748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.284720    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.385626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.385588    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.385853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.385636    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.385853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.385655    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.385853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.385674    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.385853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.385729    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.385853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.385783    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.385853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.385809    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.386205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.385860    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfg76\" (UniqueName: \"kubernetes.io/projected/3a8de801-f2f2-4b08-ad24-022a2e063bc8-kube-api-access-qfg76\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.386205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.385878    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.386205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.386154    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.386487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.386460    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.386747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.386506    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.386747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.386515    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.386747 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.386538    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.388276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.388253    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.388488 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.388469    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.411602 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.411572    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8de801-f2f2-4b08-ad24-022a2e063bc8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.411796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.411776    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfg76\" (UniqueName: \"kubernetes.io/projected/3a8de801-f2f2-4b08-ad24-022a2e063bc8-kube-api-access-qfg76\") pod \"maas-default-gateway-openshift-default-5bc884dd87-9tbj6\" (UID: \"3a8de801-f2f2-4b08-ad24-022a2e063bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.516239 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.516131    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:16.836198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.836167    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"]
Nov 02 13:39:16.836892 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:39:16.836853    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8de801_f2f2_4b08_ad24_022a2e063bc8.slice/crio-44c45342cac1888563484221a7ef13e909291bf81b9a6676efba59caca40f53e WatchSource:0}: Error finding container 44c45342cac1888563484221a7ef13e909291bf81b9a6676efba59caca40f53e: Status 404 returned error can't find the container with id 44c45342cac1888563484221a7ef13e909291bf81b9a6676efba59caca40f53e
Nov 02 13:39:16.839087 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.839052    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:39:16.839204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.839122    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:39:16.839204 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:16.839152    5879 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Nov 02 13:39:17.343932 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:17.343891    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" event={"ID":"3a8de801-f2f2-4b08-ad24-022a2e063bc8","Type":"ContainerStarted","Data":"aae08de1802f25f184133b1ab0b5c5738006e72d8739764c93f9c85fd73c5e7a"}
Nov 02 13:39:17.344459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:17.343944    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" event={"ID":"3a8de801-f2f2-4b08-ad24-022a2e063bc8","Type":"ContainerStarted","Data":"44c45342cac1888563484221a7ef13e909291bf81b9a6676efba59caca40f53e"}
Nov 02 13:39:17.516466 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:17.516431    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:17.517912 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:17.517883    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9tbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body=
Nov 02 13:39:17.518060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:17.517933    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" podUID="3a8de801-f2f2-4b08-ad24-022a2e063bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused"
Nov 02 13:39:18.517368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:18.517330    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9tbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body=
Nov 02 13:39:18.517788 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:18.517387    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" podUID="3a8de801-f2f2-4b08-ad24-022a2e063bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused"
Nov 02 13:39:19.517559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:19.517520    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9tbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body=
Nov 02 13:39:19.518037 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:19.517582    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" podUID="3a8de801-f2f2-4b08-ad24-022a2e063bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused"
Nov 02 13:39:20.517593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:20.517557    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9tbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body=
Nov 02 13:39:20.518073 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:20.517616    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" podUID="3a8de801-f2f2-4b08-ad24-022a2e063bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused"
Nov 02 13:39:21.516776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:21.516739    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9tbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body=
Nov 02 13:39:21.517052 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:21.516799    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" podUID="3a8de801-f2f2-4b08-ad24-022a2e063bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused"
Nov 02 13:39:22.516695 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:22.516661    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9tbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body=
Nov 02 13:39:22.517167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:22.516722    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" podUID="3a8de801-f2f2-4b08-ad24-022a2e063bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused"
Nov 02 13:39:23.516837 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:23.516802    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9tbj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body=
Nov 02 13:39:23.517323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:23.516861    5879 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" podUID="3a8de801-f2f2-4b08-ad24-022a2e063bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused"
Nov 02 13:39:24.989938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:24.989903    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:24.990696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:24.990669    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:24.991405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:24.991385    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6"
Nov 02 13:39:25.054209 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:25.054148    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" podStartSLOduration=9.054134905 podStartE2EDuration="9.054134905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:39:17.405228213 +0000 UTC m=+4303.511931065" watchObservedRunningTime="2026-11-02 13:39:25.054134905 +0000 UTC m=+4311.160837757"
Nov 02 13:39:42.508982 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:39:42.508940    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:39:44.820696 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:39:44.820606    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:40:00.868190 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:00.868141    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:40:00.868818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:00.868215    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:40:05.851848 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:05.851804    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:40:05.989657 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:05.989616    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:40:06.030378 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:06.030341    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"]
Nov 02 13:40:06.043449 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:06.043420    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d5a1784-6whqq"]
Nov 02 13:40:08.016070 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:08.016031    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d52089-5ede-45b2-90b8-c2e4fc32359b" path="/var/lib/kubelet/pods/30d52089-5ede-45b2-90b8-c2e4fc32359b/volumes"
Nov 02 13:40:14.820376 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:40:14.820336    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:40:38.718268 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:38.718232    5879 scope.go:122] "RemoveContainer" containerID="da21ac5db4d48b334176d6673e961ac92a99e1a557cf92099ece153527f5b844"
Nov 02 13:40:38.727502 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:38.727477    5879 scope.go:122] "RemoveContainer" containerID="bee2a9f738697ac8fe5a4710e4f3b83a66a8be2a29f1aa48ccfa7937fdd5c373"
Nov 02 13:40:41.970601 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:41.970566    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:40:42.363238 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:42.363140    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:40:42.669463 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:42.669434    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:40:44.358355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:44.358321    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:40:44.787211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:40:44.787174    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:40:44.822062 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:40:44.821987    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:41:09.727941 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:41:09.727900    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:41:14.819948 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:41:14.819862    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:41:38.754651 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:41:38.754617    5879 scope.go:122] "RemoveContainer" containerID="4ceebc075672ef47d62c57b1bdb2e0b65ce38675bfa90008039903c67e4db893"
Nov 02 13:41:42.798367 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:41:42.798323    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:41:44.458071 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:41:44.458028    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:41:44.822406 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:41:44.822320    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:41:59.191286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:41:59.191246    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:41:59.689765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:41:59.689723    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:41:59.880208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:41:59.880173    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:03.339390 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:03.339303    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:03.479041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:03.478965    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:04.848328 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:04.848262    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:04.982569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:04.982534    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:05.296310 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:05.296274    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:05.426852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:05.426798    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:06.694894 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:06.694859    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:08.402109 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:08.402051    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:10.190283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:10.190254    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:11.973923 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:11.973888    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:13.886936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:13.886896    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:14.820788 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:42:14.820749    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:42:15.720969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:15.720930    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:17.377355 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:17.377321    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:18.116590 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:18.116549    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:18.719409 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:18.719374    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:18.874853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:18.874809    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:19.098766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:19.098690    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:19.243903 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:19.243868    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:19.955069 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:19.955035    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"]
Nov 02 13:42:19.955853 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:19.955833    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d9f2c8-011b-49fa-9f7b-5a36ee074541" containerName="istio-proxy"
Nov 02 13:42:19.959821 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:19.959793    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:19.977741 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:19.977707    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f119d1\"/\"openshift-service-ca.crt\""
Nov 02 13:42:19.991667 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:19.991630    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f119d1\"/\"kube-root-ca.crt\""
Nov 02 13:42:19.991852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:19.991755    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f119d1\"/\"default-dockercfg-pnc9h\""
Nov 02 13:42:19.992399 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:19.992371    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f119d1\"/\"simulator-single-main-f119d1-kserve-self-signed-certs\""
Nov 02 13:42:19.993943 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:19.993921    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"]
Nov 02 13:42:20.022612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.022577    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-model-cache\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.022612 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.022630    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-home\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.022911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.022708    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tnhb\" (UniqueName: \"kubernetes.io/projected/5fbb71fb-8795-4e01-a19d-81efff6de06a-kube-api-access-7tnhb\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.022911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.022760    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbb71fb-8795-4e01-a19d-81efff6de06a-tls-certs\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.022911 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.022899    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-dshm\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.023151 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.022924    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-tmp-dir\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.123654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.123612    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-dshm\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.123654 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.123653    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-tmp-dir\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.123938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.123723    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-model-cache\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.123938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.123776    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-home\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.123938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.123808    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tnhb\" (UniqueName: \"kubernetes.io/projected/5fbb71fb-8795-4e01-a19d-81efff6de06a-kube-api-access-7tnhb\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.123938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.123848    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbb71fb-8795-4e01-a19d-81efff6de06a-tls-certs\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.124269 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.124242    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-tmp-dir\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.124440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.124407    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-home\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.124652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.124632    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-model-cache\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.124702 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.124670    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:20.126246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.126214    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-dshm\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.126534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.126517    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbb71fb-8795-4e01-a19d-81efff6de06a-tls-certs\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.149393 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.149362    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tnhb\" (UniqueName: \"kubernetes.io/projected/5fbb71fb-8795-4e01-a19d-81efff6de06a-kube-api-access-7tnhb\") pod \"simulator-single-main-f119d1-kserve-598689f746-llkpf\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") " pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.274253 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.274157    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:20.578337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:20.578304    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"]
Nov 02 13:42:20.578572 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:42:20.578547    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fbb71fb_8795_4e01_a19d_81efff6de06a.slice/crio-1bf2c29fd36d475fb16a067d15ef7a00fa2ce20c845b2e2c1aff1406d4139cc1 WatchSource:0}: Error finding container 1bf2c29fd36d475fb16a067d15ef7a00fa2ce20c845b2e2c1aff1406d4139cc1: Status 404 returned error can't find the container with id 1bf2c29fd36d475fb16a067d15ef7a00fa2ce20c845b2e2c1aff1406d4139cc1
Nov 02 13:42:21.074906 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:21.074862    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:21.446551 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:21.446511    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf" event={"ID":"5fbb71fb-8795-4e01-a19d-81efff6de06a","Type":"ContainerStarted","Data":"545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b"}
Nov 02 13:42:21.446790 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:21.446559    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf" event={"ID":"5fbb71fb-8795-4e01-a19d-81efff6de06a","Type":"ContainerStarted","Data":"1bf2c29fd36d475fb16a067d15ef7a00fa2ce20c845b2e2c1aff1406d4139cc1"}
Nov 02 13:42:21.502368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:21.502310    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf" podStartSLOduration=2.502294599 podStartE2EDuration="2.502294599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:42:21.50053655 +0000 UTC m=+4487.607239400" watchObservedRunningTime="2026-11-02 13:42:21.502294599 +0000 UTC m=+4487.608997450"
Nov 02 13:42:22.274440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:22.274395    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:22.888459 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:22.888424    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:24.690575 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:24.690541    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:26.483738 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:26.483700    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:28.178607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:28.178562    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:30.028622 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:30.028583    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:30.274697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:30.274659    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:30.287824 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:30.287730    5879 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:30.507013 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:30.506963    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:42:31.774507 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:31.774459    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:33.477970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:33.477927    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:35.178245 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:35.178210    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:37.034287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:37.034243    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:38.788825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:38.788794    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:40.614358 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:40.614324    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:42.288535 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:42.288501    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:42.912051 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:42.912016    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:44.126520 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:44.126413    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:44.821866 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:42:44.821809    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:42:45.877829 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:45.877791    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:47.681968 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:47.681921    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:49.493647 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:49.493599    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:51.267970 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:51.267935    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:53.071392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:53.071350    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:54.962852 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:54.962812    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:56.570714 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:56.570678    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:58.269196 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:58.269155    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:42:59.976759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:42:59.976719    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:01.325431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:01.325393    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:01.722309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:01.722275    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:01.876144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:01.876104    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:03.480637 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:03.480602    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:05.277179 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:05.277144    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:06.234885 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:06.234846    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"]
Nov 02 13:43:06.345376 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:06.345334    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:07.094823 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:07.094762    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:08.379704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:08.379673    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:08.516249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:08.516218    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:08.658549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:08.658461    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:08.805094 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:08.805057    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:14.555166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:14.555130    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:14.725666 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:14.725636    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:14.821373 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:43:14.821265    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:43:21.256056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.255984    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf" podUID="5fbb71fb-8795-4e01-a19d-81efff6de06a" containerName="main" containerID="cri-o://545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b" gracePeriod=45
Nov 02 13:43:21.517458 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.517380    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:43:21.585584 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.585549    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-tmp-dir\") pod \"5fbb71fb-8795-4e01-a19d-81efff6de06a\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") "
Nov 02 13:43:21.585779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.585614    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-model-cache\") pod \"5fbb71fb-8795-4e01-a19d-81efff6de06a\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") "
Nov 02 13:43:21.585779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.585686    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fbb71fb-8795-4e01-a19d-81efff6de06a-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbb71fb-8795-4e01-a19d-81efff6de06a-tls-certs\") pod \"5fbb71fb-8795-4e01-a19d-81efff6de06a\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") "
Nov 02 13:43:21.585779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.585742    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-home\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-home\") pod \"5fbb71fb-8795-4e01-a19d-81efff6de06a\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") "
Nov 02 13:43:21.585779 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.585773    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fbb71fb-8795-4e01-a19d-81efff6de06a-kube-api-access-7tnhb\" (UniqueName: \"kubernetes.io/projected/5fbb71fb-8795-4e01-a19d-81efff6de06a-kube-api-access-7tnhb\") pod \"5fbb71fb-8795-4e01-a19d-81efff6de06a\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") "
Nov 02 13:43:21.586049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.585815    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-dshm\") pod \"5fbb71fb-8795-4e01-a19d-81efff6de06a\" (UID: \"5fbb71fb-8795-4e01-a19d-81efff6de06a\") "
Nov 02 13:43:21.586049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.585890    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-tmp-dir" pod "5fbb71fb-8795-4e01-a19d-81efff6de06a" (UID: "5fbb71fb-8795-4e01-a19d-81efff6de06a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:43:21.586049 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.585953    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-model-cache" pod "5fbb71fb-8795-4e01-a19d-81efff6de06a" (UID: "5fbb71fb-8795-4e01-a19d-81efff6de06a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:43:21.586246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.586091    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-home" pod "5fbb71fb-8795-4e01-a19d-81efff6de06a" (UID: "5fbb71fb-8795-4e01-a19d-81efff6de06a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:43:21.586343 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.586320    5879 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-home\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:43:21.586405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.586375    5879 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-tmp-dir\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:43:21.586405 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.586394    5879 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-model-cache\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:43:21.589036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.588315    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-dshm" pod "5fbb71fb-8795-4e01-a19d-81efff6de06a" (UID: "5fbb71fb-8795-4e01-a19d-81efff6de06a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 13:43:21.589036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.588377    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fbb71fb-8795-4e01-a19d-81efff6de06a-tls-certs" pod "5fbb71fb-8795-4e01-a19d-81efff6de06a" (UID: "5fbb71fb-8795-4e01-a19d-81efff6de06a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:43:21.589036 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.588631    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbb71fb-8795-4e01-a19d-81efff6de06a-kube-api-access-7tnhb" pod "5fbb71fb-8795-4e01-a19d-81efff6de06a" (UID: "5fbb71fb-8795-4e01-a19d-81efff6de06a"). InnerVolumeSpecName "kube-api-access-7tnhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:43:21.687359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.687324    5879 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbb71fb-8795-4e01-a19d-81efff6de06a-tls-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:43:21.687359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.687355    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tnhb\" (UniqueName: \"kubernetes.io/projected/5fbb71fb-8795-4e01-a19d-81efff6de06a-kube-api-access-7tnhb\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:43:21.687359 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.687365    5879 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbb71fb-8795-4e01-a19d-81efff6de06a-dshm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:43:21.750690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.750653    5879 generic.go:356] "Generic (PLEG): container finished" podID="5fbb71fb-8795-4e01-a19d-81efff6de06a" containerID="545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b" exitCode=0
Nov 02 13:43:21.750849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.750697    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"
Nov 02 13:43:21.750849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.750702    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf" event={"ID":"5fbb71fb-8795-4e01-a19d-81efff6de06a","Type":"ContainerDied","Data":"545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b"}
Nov 02 13:43:21.750849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.750733    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf" event={"ID":"5fbb71fb-8795-4e01-a19d-81efff6de06a","Type":"ContainerDied","Data":"1bf2c29fd36d475fb16a067d15ef7a00fa2ce20c845b2e2c1aff1406d4139cc1"}
Nov 02 13:43:21.750849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.750749    5879 scope.go:122] "RemoveContainer" containerID="545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b"
Nov 02 13:43:21.763897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.763878    5879 scope.go:122] "RemoveContainer" containerID="545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b"
Nov 02 13:43:21.764179 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:43:21.764155    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b\": container with ID starting with 545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b not found: ID does not exist" containerID="545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b"
Nov 02 13:43:21.764246 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.764189    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b"} err="failed to get container status \"545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b\": rpc error: code = NotFound desc = could not find container \"545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b\": container with ID starting with 545d6af1fb783e1f1e80717f914e30304094b5185be92bc946686d9dac8de05b not found: ID does not exist"
Nov 02 13:43:21.832965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.832881    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"]
Nov 02 13:43:21.845143 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:21.845110    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf"]
Nov 02 13:43:22.015064 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:22.015032    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fbb71fb-8795-4e01-a19d-81efff6de06a" path="/var/lib/kubelet/pods/5fbb71fb-8795-4e01-a19d-81efff6de06a/volumes"
Nov 02 13:43:22.496052 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:22.496007    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-f119d1/simulator-single-main-f119d1-kserve-598689f746-llkpf" podUID="5fbb71fb-8795-4e01-a19d-81efff6de06a" containerName="main" probeResult="failure" output="Get \"https://10.134.1.137:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:43:35.771236 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:35.771191    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:41.279309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:41.279271    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:41.483915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:41.483884    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:42.874125 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:42.874083    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:43.074360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:43.074312    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:43.273887 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:43.273797    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:44.597536 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:44.597501    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:44.820268 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:43:44.820230    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:43:45.147726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:45.147678    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:46.061815 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:46.061777    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:47.029348 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:47.029305    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:48.901082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:48.901046    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:50.709696 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:50.709661    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:52.497167 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:52.497120    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:52.909805 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:52.909739    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:53.086174 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:53.086143    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:54.531814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:54.531782    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:56.287579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:56.287539    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:58.086828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:58.086788    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:43:59.789235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:43:59.789195    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:01.484199 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:01.484118    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:02.924744 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:02.924708    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:03.148346 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:03.148315    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:03.342082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:03.341972    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:04.892865 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:04.892827    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:05.367776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:05.367740    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:05.545564 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:05.545532    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:05.948118 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:05.948079    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:11.264385 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:11.264326    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:14.332077 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:14.331835    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:14.548029 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:14.547850    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:14.867135 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:44:14.867066    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:44:15.866208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:15.866161    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:44:15.866688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:15.866239    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:44:27.160045 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:27.160012    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:40.035843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:40.035809    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:43.158822 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:43.158783    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:44.819894 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:44:44.819859    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:44:55.493127 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:55.493092    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:44:59.471137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:44:59.471095    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:45:14.820175 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:45:14.820137    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:45:27.132905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:45:27.132865    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:45:27.288254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:45:27.288220    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:45:41.947979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:45:41.947942    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:45:43.283132 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:45:43.283098    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:45:44.821702 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:45:44.821662    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:46:14.821027 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:46:14.820971    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:46:22.102576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:46:22.102537    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:46:22.287147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:46:22.287113    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:46:23.025609 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:46:23.025577    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:46:43.409372 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:46:43.409333    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:46:44.821605 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:46:44.821568    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:47:02.257235 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:47:02.257192    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:47:14.820369 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:47:14.820281    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:47:18.977092 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:47:18.977063    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:47:43.541368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:47:43.541328    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:47:44.820796 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:47:44.820759    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:48:05.096532 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:48:05.096502    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:48:05.246443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:48:05.246406    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:48:05.406553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:48:05.406513    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:48:08.472115 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:48:08.472042    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-27twq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:48:08.472565 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:48:08.472171    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podUID="c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:48:14.872580 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:48:14.872537    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:48:15.858182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:48:15.858139    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:48:15.858432 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:48:15.858397    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:48:43.665893 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:48:43.665802    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:48:44.820124 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:48:44.820090    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:48:53.504017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:48:53.503098    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-27twq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:48:53.504017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:48:53.503186    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-27twq" podUID="c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:49:14.821920 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:49:14.821883    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:49:43.902025 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:49:43.894786    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:49:44.868777 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:49:44.868726    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:50:14.820555 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:50:14.820461    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:50:44.820582 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:50:44.820542    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:50:45.145396 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:50:45.145362    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:51:14.821506 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:51:14.821388    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:51:44.100035 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:51:44.099893    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:51:44.230646 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:51:44.230612    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:51:44.825715 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:51:44.825666    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:52:10.223765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:10.223712    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:14.337934 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:14.337900    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:14.830656 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:52:14.830617    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:52:14.909597 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:14.909542    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:15.184041 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:15.183974    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:18.744413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:18.744363    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:18.908577 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:18.908537    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:20.418219 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:20.418181    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:20.597114 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:20.597074    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:20.792896 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:20.792800    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:22.489596 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:22.489533    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:24.377837 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:24.377802    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:26.189176 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:26.189147    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:28.110469 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:28.110423    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:29.932061 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:29.932020    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:31.966681 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:31.966642    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:33.582900 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:33.582865    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:33.763156 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:33.763111    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:34.212558 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:34.212522    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:34.365588 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:34.365546    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:34.515767 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:34.515666    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:35.782858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:35.782820    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:36.346166 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:36.346129    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:37.707247 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:37.707211    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:39.623267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:39.623223    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:41.368065 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:41.368029    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:43.170616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:43.170569    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:44.237356 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:44.237323    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:44.820197 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:52:44.820159    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:52:44.868874 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:44.868839    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:46.676910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:46.676869    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:48.585068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:48.584986    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:50.377772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:50.377706    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:52.101842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:52.101796    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:53.869300 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:53.869261    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:55.808918 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:55.808873    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:57.739454 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:57.739414    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:52:59.484468 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:52:59.484427    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:01.283548 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:01.283515    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:03.072520 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:03.072483    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:04.877406 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:04.877356    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:06.581804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:06.581756    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:08.372763 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:08.372725    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:10.179239 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:10.179199    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:11.970279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:11.970239    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:13.788595 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:13.788497    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:14.348642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:14.348601    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:14.821274 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:53:14.821230    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:53:15.585704 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:15.585660    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:17.297697 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:17.297660    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:19.020309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:19.020262    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:19.144716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:19.144680    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:19.484144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:19.484107    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:20.783474 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:20.783436    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:22.769093 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:22.769061    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:24.037671 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:24.037599    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:24.589257 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:24.589218    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:25.452527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:25.452489    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:25.579135 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:25.579098    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:26.224660 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:26.224607    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:26.664936 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:26.664901    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:32.342203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:32.342168    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:32.481424 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:32.481384    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:44.367913 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:44.367844    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:44.820743 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:53:44.820655    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:53:56.888337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:56.888293    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:57.054889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:57.054851    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:58.398938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:58.398898    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:58.559842 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:58.559798    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:58.710027 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:58.709928    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:53:58.879345 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:53:58.879302    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:00.296048 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:00.296012    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:01.530160 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:01.530126    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:02.208814 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:02.208779    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:02.337434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:02.337384    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:04.275253 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:04.275220    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:06.075566 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:06.075530    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:07.936897 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:07.936857    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:08.668634 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:08.668594    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:08.802416 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:08.802369    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:09.770016 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:09.769955    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:11.517784 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:11.517744    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:13.367570 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:13.367536    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:14.821013 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:54:14.820963    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:54:15.162248 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:15.162209    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:16.964271 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:16.964233    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:18.316394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:18.316360    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:18.445607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:18.445563    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:18.697155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:18.697119    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:20.243626 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:20.243592    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:20.901249 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:20.901216    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:21.016460 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:21.016422    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:21.414038 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:21.413984    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:44.477439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:54:44.477341    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:54:44.821015 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:54:44.820922    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:55:14.822026 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:55:14.821953    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:55:44.667302 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:55:44.667260    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:55:44.821136 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:55:44.821095    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:56:14.821072 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:56:14.821035    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:56:26.032203 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:26.032155    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9tbj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:56:26.032830 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:26.032246    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" podUID="3a8de801-f2f2-4b08-ad24-022a2e063bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:56:44.723630 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:44.723590    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:56:44.820975 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:56:44.820935    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:56:51.759360 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:51.759321    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"]
Nov 02 13:56:51.760352 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:51.760274    5879 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r" podUID="19d1d69f-c28b-492f-8ffc-4ea1ed28cfce" containerName="maas-api" containerID="cri-o://cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3" gracePeriod=30
Nov 02 13:56:52.501170 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:52.501115    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r" podUID="19d1d69f-c28b-492f-8ffc-4ea1ed28cfce" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.38:8443/health\": dial tcp 10.134.0.38:8443: connect: connection refused"
Nov 02 13:56:53.097639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.097560    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 13:56:53.190128 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.190097    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-maas-api-tls\") pod \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") "
Nov 02 13:56:53.190336 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.190289    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-kube-api-access-wwhjp\" (UniqueName: \"kubernetes.io/projected/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-kube-api-access-wwhjp\") pod \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") "
Nov 02 13:56:53.190392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.190333    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-openshift-service-ca\") pod \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") "
Nov 02 13:56:53.190447 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.190400    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-metrics-certs\") pod \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\" (UID: \"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce\") "
Nov 02 13:56:53.190731 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.190709    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-openshift-service-ca" pod "19d1d69f-c28b-492f-8ffc-4ea1ed28cfce" (UID: "19d1d69f-c28b-492f-8ffc-4ea1ed28cfce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:53.190816 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.190795    5879 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-openshift-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:56:53.192475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.192448    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-maas-api-tls" pod "19d1d69f-c28b-492f-8ffc-4ea1ed28cfce" (UID: "19d1d69f-c28b-492f-8ffc-4ea1ed28cfce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:53.192587 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.192567    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-metrics-certs" pod "19d1d69f-c28b-492f-8ffc-4ea1ed28cfce" (UID: "19d1d69f-c28b-492f-8ffc-4ea1ed28cfce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:53.192652 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.192617    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-kube-api-access-wwhjp" pod "19d1d69f-c28b-492f-8ffc-4ea1ed28cfce" (UID: "19d1d69f-c28b-492f-8ffc-4ea1ed28cfce"). InnerVolumeSpecName "kube-api-access-wwhjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:53.274123 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.274087    5879 generic.go:356] "Generic (PLEG): container finished" podID="19d1d69f-c28b-492f-8ffc-4ea1ed28cfce" containerID="cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3" exitCode=0
Nov 02 13:56:53.274337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.274131    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"
Nov 02 13:56:53.274337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.274165    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r" event={"ID":"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce","Type":"ContainerDied","Data":"cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3"}
Nov 02 13:56:53.274337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.274217    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r" event={"ID":"19d1d69f-c28b-492f-8ffc-4ea1ed28cfce","Type":"ContainerDied","Data":"fc5cdc98f5bc5d4405ede80b4c776e46ab42c052cdee60d6a8ba69767adc4b5f"}
Nov 02 13:56:53.274337 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.274241    5879 scope.go:122] "RemoveContainer" containerID="cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3"
Nov 02 13:56:53.284808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.284784    5879 scope.go:122] "RemoveContainer" containerID="cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3"
Nov 02 13:56:53.285133 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:56:53.285112    5879 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3\": container with ID starting with cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3 not found: ID does not exist" containerID="cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3"
Nov 02 13:56:53.285197 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.285143    5879 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3"} err="failed to get container status \"cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3\": rpc error: code = NotFound desc = could not find container \"cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3\": container with ID starting with cce59edfe9692e35c3d689b19057e55709df0ad6b32f70bb65d696a0b018cea3 not found: ID does not exist"
Nov 02 13:56:53.291434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.291407    5879 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-metrics-certs\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:56:53.291434 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.291435    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-maas-api-tls\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:56:53.291549 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.291445    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhjp\" (UniqueName: \"kubernetes.io/projected/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce-kube-api-access-wwhjp\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:56:53.329766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.329731    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"]
Nov 02 13:56:53.336772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:53.336741    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-65f6775b5-8jg8r"]
Nov 02 13:56:54.015979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:54.015946    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d1d69f-c28b-492f-8ffc-4ea1ed28cfce" path="/var/lib/kubelet/pods/19d1d69f-c28b-492f-8ffc-4ea1ed28cfce/volumes"
Nov 02 13:56:54.932319 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:54.932280    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"]
Nov 02 13:56:54.932759 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:54.932745    5879 cpu_manager.go:414] "containerMap: removing container" podUID="5fbb71fb-8795-4e01-a19d-81efff6de06a" containerName="main"
Nov 02 13:56:54.932807 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:54.932760    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fbb71fb-8795-4e01-a19d-81efff6de06a" containerName="main"
Nov 02 13:56:54.932849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:54.932810    5879 cpu_manager.go:414] "containerMap: removing container" podUID="19d1d69f-c28b-492f-8ffc-4ea1ed28cfce" containerName="maas-api"
Nov 02 13:56:54.932849 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:54.932817    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d1d69f-c28b-492f-8ffc-4ea1ed28cfce" containerName="maas-api"
Nov 02 13:56:54.932916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:54.932904    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d1d69f-c28b-492f-8ffc-4ea1ed28cfce" containerName="maas-api"
Nov 02 13:56:54.932916 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:54.932915    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fbb71fb-8795-4e01-a19d-81efff6de06a" containerName="main"
Nov 02 13:56:54.937581 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:54.937560    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:54.955060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:54.955034    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:56:54.969124 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:54.969094    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:56:54.970583 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:54.970564    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"]
Nov 02 13:56:55.008457 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:55.008424    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxzl\" (UniqueName: \"kubernetes.io/projected/460dd41d-abbd-43ee-a179-066dd1920fbb-kube-api-access-6zxzl\") pod \"maas-api-revoke-sub-d60ead89-xtgqz\" (UID: \"460dd41d-abbd-43ee-a179-066dd1920fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:55.008648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:55.008559    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60ead89-xtgqz\" (UID: \"460dd41d-abbd-43ee-a179-066dd1920fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:55.008648 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:55.008622    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60ead89-xtgqz\" (UID: \"460dd41d-abbd-43ee-a179-066dd1920fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:55.109838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:55.109795    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60ead89-xtgqz\" (UID: \"460dd41d-abbd-43ee-a179-066dd1920fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:55.110060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:55.109865    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60ead89-xtgqz\" (UID: \"460dd41d-abbd-43ee-a179-066dd1920fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:55.110060 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:55.109921    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxzl\" (UniqueName: \"kubernetes.io/projected/460dd41d-abbd-43ee-a179-066dd1920fbb-kube-api-access-6zxzl\") pod \"maas-api-revoke-sub-d60ead89-xtgqz\" (UID: \"460dd41d-abbd-43ee-a179-066dd1920fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:55.110622 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:55.110600    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d60ead89-xtgqz\" (UID: \"460dd41d-abbd-43ee-a179-066dd1920fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:55.112544 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:55.112526    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d60ead89-xtgqz\" (UID: \"460dd41d-abbd-43ee-a179-066dd1920fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:55.135848 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:55.135806    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxzl\" (UniqueName: \"kubernetes.io/projected/460dd41d-abbd-43ee-a179-066dd1920fbb-kube-api-access-6zxzl\") pod \"maas-api-revoke-sub-d60ead89-xtgqz\" (UID: \"460dd41d-abbd-43ee-a179-066dd1920fbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:55.250889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:55.250794    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:55.563644 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:55.563614    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"]
Nov 02 13:56:55.564204 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:56:55.564179    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460dd41d_abbd_43ee_a179_066dd1920fbb.slice/crio-4078d1f1082dec6aca15b433728cc5e7874ec53bd957c197b33bced97cdec381 WatchSource:0}: Error finding container 4078d1f1082dec6aca15b433728cc5e7874ec53bd957c197b33bced97cdec381: Status 404 returned error can't find the container with id 4078d1f1082dec6aca15b433728cc5e7874ec53bd957c197b33bced97cdec381
Nov 02 13:56:56.297054 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:56.296926    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz" event={"ID":"460dd41d-abbd-43ee-a179-066dd1920fbb","Type":"ContainerStarted","Data":"c8b968ff9932237cb082081931e6beb2ea78698ad3701343badc26c7a775ff87"}
Nov 02 13:56:56.297054 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:56.296977    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz" event={"ID":"460dd41d-abbd-43ee-a179-066dd1920fbb","Type":"ContainerStarted","Data":"4078d1f1082dec6aca15b433728cc5e7874ec53bd957c197b33bced97cdec381"}
Nov 02 13:56:56.342462 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:56.342392    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz" podStartSLOduration=2.342375626 podStartE2EDuration="2.342375626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:56:56.340927868 +0000 UTC m=+5362.447630722" watchObservedRunningTime="2026-11-02 13:56:56.342375626 +0000 UTC m=+5362.449078477"
Nov 02 13:56:57.303368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:57.303332    5879 generic.go:356] "Generic (PLEG): container finished" podID="460dd41d-abbd-43ee-a179-066dd1920fbb" containerID="c8b968ff9932237cb082081931e6beb2ea78698ad3701343badc26c7a775ff87" exitCode=0
Nov 02 13:56:57.303797 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:57.303422    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz" event={"ID":"460dd41d-abbd-43ee-a179-066dd1920fbb","Type":"ContainerDied","Data":"c8b968ff9932237cb082081931e6beb2ea78698ad3701343badc26c7a775ff87"}
Nov 02 13:56:58.454058 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:58.454025    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:58.550308 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:58.550273    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/460dd41d-abbd-43ee-a179-066dd1920fbb-kube-api-access-6zxzl\" (UniqueName: \"kubernetes.io/projected/460dd41d-abbd-43ee-a179-066dd1920fbb-kube-api-access-6zxzl\") pod \"460dd41d-abbd-43ee-a179-066dd1920fbb\" (UID: \"460dd41d-abbd-43ee-a179-066dd1920fbb\") "
Nov 02 13:56:58.550487 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:58.550343    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-xks-ca\") pod \"460dd41d-abbd-43ee-a179-066dd1920fbb\" (UID: \"460dd41d-abbd-43ee-a179-066dd1920fbb\") "
Nov 02 13:56:58.550553 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:58.550507    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-openshift-ca\") pod \"460dd41d-abbd-43ee-a179-066dd1920fbb\" (UID: \"460dd41d-abbd-43ee-a179-066dd1920fbb\") "
Nov 02 13:56:58.550872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:58.550846    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-openshift-ca" pod "460dd41d-abbd-43ee-a179-066dd1920fbb" (UID: "460dd41d-abbd-43ee-a179-066dd1920fbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:56:58.552828 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:58.552803    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-xks-ca" pod "460dd41d-abbd-43ee-a179-066dd1920fbb" (UID: "460dd41d-abbd-43ee-a179-066dd1920fbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:56:58.552927 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:58.552825    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460dd41d-abbd-43ee-a179-066dd1920fbb-kube-api-access-6zxzl" pod "460dd41d-abbd-43ee-a179-066dd1920fbb" (UID: "460dd41d-abbd-43ee-a179-066dd1920fbb"). InnerVolumeSpecName "kube-api-access-6zxzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:56:58.652357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:58.652319    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:56:58.652357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:58.652350    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxzl\" (UniqueName: \"kubernetes.io/projected/460dd41d-abbd-43ee-a179-066dd1920fbb-kube-api-access-6zxzl\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:56:58.652357 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:58.652361    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/460dd41d-abbd-43ee-a179-066dd1920fbb-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:56:59.314629 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:59.314596    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"
Nov 02 13:56:59.314818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:59.314632    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz" event={"ID":"460dd41d-abbd-43ee-a179-066dd1920fbb","Type":"ContainerDied","Data":"4078d1f1082dec6aca15b433728cc5e7874ec53bd957c197b33bced97cdec381"}
Nov 02 13:56:59.314818 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:56:59.314683    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4078d1f1082dec6aca15b433728cc5e7874ec53bd957c197b33bced97cdec381"
Nov 02 13:57:05.304280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.304245    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"]
Nov 02 13:57:05.304693 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.304679    5879 cpu_manager.go:414] "containerMap: removing container" podUID="460dd41d-abbd-43ee-a179-066dd1920fbb" containerName="revoke-keys"
Nov 02 13:57:05.304693 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.304689    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460dd41d-abbd-43ee-a179-066dd1920fbb" containerName="revoke-keys"
Nov 02 13:57:05.304776 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.304770    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="460dd41d-abbd-43ee-a179-066dd1920fbb" containerName="revoke-keys"
Nov 02 13:57:05.308159 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.308141    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:05.329843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.329805    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"]
Nov 02 13:57:05.341281 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.341248    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:57:05.341485 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.341310    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:57:05.430974 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.430936    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0692b79-56tjj\" (UID: \"1240f278-eefe-4caa-92a5-e07818114e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:05.431173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.430987    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0692b79-56tjj\" (UID: \"1240f278-eefe-4caa-92a5-e07818114e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:05.431173 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.431134    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dfh9\" (UniqueName: \"kubernetes.io/projected/1240f278-eefe-4caa-92a5-e07818114e29-kube-api-access-6dfh9\") pod \"maas-api-revoke-sub-a0692b79-56tjj\" (UID: \"1240f278-eefe-4caa-92a5-e07818114e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:05.532766 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.532728    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0692b79-56tjj\" (UID: \"1240f278-eefe-4caa-92a5-e07818114e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:05.532939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.532779    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0692b79-56tjj\" (UID: \"1240f278-eefe-4caa-92a5-e07818114e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:05.532939 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.532810    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dfh9\" (UniqueName: \"kubernetes.io/projected/1240f278-eefe-4caa-92a5-e07818114e29-kube-api-access-6dfh9\") pod \"maas-api-revoke-sub-a0692b79-56tjj\" (UID: \"1240f278-eefe-4caa-92a5-e07818114e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:05.533541 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.533519    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0692b79-56tjj\" (UID: \"1240f278-eefe-4caa-92a5-e07818114e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:05.535320 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.535288    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0692b79-56tjj\" (UID: \"1240f278-eefe-4caa-92a5-e07818114e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:05.558589 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.558506    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dfh9\" (UniqueName: \"kubernetes.io/projected/1240f278-eefe-4caa-92a5-e07818114e29-kube-api-access-6dfh9\") pod \"maas-api-revoke-sub-a0692b79-56tjj\" (UID: \"1240f278-eefe-4caa-92a5-e07818114e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:05.621985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.621945    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:05.925431 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:05.925398    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"]
Nov 02 13:57:05.926467 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:57:05.926439    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1240f278_eefe_4caa_92a5_e07818114e29.slice/crio-8689686ecb3f5eeba2b26db3c2d324da60ab6065d0185d109fc97d5b8636c7f5 WatchSource:0}: Error finding container 8689686ecb3f5eeba2b26db3c2d324da60ab6065d0185d109fc97d5b8636c7f5: Status 404 returned error can't find the container with id 8689686ecb3f5eeba2b26db3c2d324da60ab6065d0185d109fc97d5b8636c7f5
Nov 02 13:57:06.351550 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:06.351515    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj" event={"ID":"1240f278-eefe-4caa-92a5-e07818114e29","Type":"ContainerStarted","Data":"82ff33acaaa28e1cbdd5f929dd007ca3c9d12fa6733129cdfacac2d4ff9fb4fa"}
Nov 02 13:57:06.351965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:06.351566    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj" event={"ID":"1240f278-eefe-4caa-92a5-e07818114e29","Type":"ContainerStarted","Data":"8689686ecb3f5eeba2b26db3c2d324da60ab6065d0185d109fc97d5b8636c7f5"}
Nov 02 13:57:06.397884 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:06.397826    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj" podStartSLOduration=1.39780932 podStartE2EDuration="1.39780932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:57:06.396887527 +0000 UTC m=+5372.503590371" watchObservedRunningTime="2026-11-02 13:57:06.39780932 +0000 UTC m=+5372.504512171"
Nov 02 13:57:07.358368 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:07.358325    5879 generic.go:356] "Generic (PLEG): container finished" podID="1240f278-eefe-4caa-92a5-e07818114e29" containerID="82ff33acaaa28e1cbdd5f929dd007ca3c9d12fa6733129cdfacac2d4ff9fb4fa" exitCode=0
Nov 02 13:57:07.358808 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:07.358374    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj" event={"ID":"1240f278-eefe-4caa-92a5-e07818114e29","Type":"ContainerDied","Data":"82ff33acaaa28e1cbdd5f929dd007ca3c9d12fa6733129cdfacac2d4ff9fb4fa"}
Nov 02 13:57:08.506707 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:08.506677    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:08.662979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:08.662935    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-xks-ca\") pod \"1240f278-eefe-4caa-92a5-e07818114e29\" (UID: \"1240f278-eefe-4caa-92a5-e07818114e29\") "
Nov 02 13:57:08.663201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:08.663062    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-openshift-ca\") pod \"1240f278-eefe-4caa-92a5-e07818114e29\" (UID: \"1240f278-eefe-4caa-92a5-e07818114e29\") "
Nov 02 13:57:08.663201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:08.663172    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1240f278-eefe-4caa-92a5-e07818114e29-kube-api-access-6dfh9\" (UniqueName: \"kubernetes.io/projected/1240f278-eefe-4caa-92a5-e07818114e29-kube-api-access-6dfh9\") pod \"1240f278-eefe-4caa-92a5-e07818114e29\" (UID: \"1240f278-eefe-4caa-92a5-e07818114e29\") "
Nov 02 13:57:08.663467 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:08.663439    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-openshift-ca" pod "1240f278-eefe-4caa-92a5-e07818114e29" (UID: "1240f278-eefe-4caa-92a5-e07818114e29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:08.665496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:08.665468    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-xks-ca" pod "1240f278-eefe-4caa-92a5-e07818114e29" (UID: "1240f278-eefe-4caa-92a5-e07818114e29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:57:08.665627 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:08.665517    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1240f278-eefe-4caa-92a5-e07818114e29-kube-api-access-6dfh9" pod "1240f278-eefe-4caa-92a5-e07818114e29" (UID: "1240f278-eefe-4caa-92a5-e07818114e29"). InnerVolumeSpecName "kube-api-access-6dfh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:08.764755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:08.764717    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:57:08.764755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:08.764750    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1240f278-eefe-4caa-92a5-e07818114e29-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:57:08.764755 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:08.764762    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dfh9\" (UniqueName: \"kubernetes.io/projected/1240f278-eefe-4caa-92a5-e07818114e29-kube-api-access-6dfh9\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:57:09.375484 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:09.375286    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj" event={"ID":"1240f278-eefe-4caa-92a5-e07818114e29","Type":"ContainerDied","Data":"8689686ecb3f5eeba2b26db3c2d324da60ab6065d0185d109fc97d5b8636c7f5"}
Nov 02 13:57:09.375677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:09.375534    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"
Nov 02 13:57:09.375677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:09.375545    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8689686ecb3f5eeba2b26db3c2d324da60ab6065d0185d109fc97d5b8636c7f5"
Nov 02 13:57:14.821863 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:57:14.821823    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:57:27.056475 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.056436    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"]
Nov 02 13:57:27.056971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.056909    5879 cpu_manager.go:414] "containerMap: removing container" podUID="1240f278-eefe-4caa-92a5-e07818114e29" containerName="revoke-keys"
Nov 02 13:57:27.056971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.056921    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1240f278-eefe-4caa-92a5-e07818114e29" containerName="revoke-keys"
Nov 02 13:57:27.057088 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.057041    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="1240f278-eefe-4caa-92a5-e07818114e29" containerName="revoke-keys"
Nov 02 13:57:27.060370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.060350    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:27.082787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.082748    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2135802a-chnnp\" (UID: \"f22c82c7-938a-47d9-bf58-6d204d63794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:27.082787 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.082810    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w64gn\" (UniqueName: \"kubernetes.io/projected/f22c82c7-938a-47d9-bf58-6d204d63794f-kube-api-access-w64gn\") pod \"maas-api-revoke-sub-2135802a-chnnp\" (UID: \"f22c82c7-938a-47d9-bf58-6d204d63794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:27.083188 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.082921    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2135802a-chnnp\" (UID: \"f22c82c7-938a-47d9-bf58-6d204d63794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:27.085097 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.085064    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"]
Nov 02 13:57:27.096928 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.096898    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:57:27.097169 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.096957    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:57:27.184490 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.184457    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2135802a-chnnp\" (UID: \"f22c82c7-938a-47d9-bf58-6d204d63794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:27.184691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.184579    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2135802a-chnnp\" (UID: \"f22c82c7-938a-47d9-bf58-6d204d63794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:27.184691 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.184618    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w64gn\" (UniqueName: \"kubernetes.io/projected/f22c82c7-938a-47d9-bf58-6d204d63794f-kube-api-access-w64gn\") pod \"maas-api-revoke-sub-2135802a-chnnp\" (UID: \"f22c82c7-938a-47d9-bf58-6d204d63794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:27.185304 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.185224    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2135802a-chnnp\" (UID: \"f22c82c7-938a-47d9-bf58-6d204d63794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:27.187189 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.187161    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2135802a-chnnp\" (UID: \"f22c82c7-938a-47d9-bf58-6d204d63794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:27.211286 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.211252    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w64gn\" (UniqueName: \"kubernetes.io/projected/f22c82c7-938a-47d9-bf58-6d204d63794f-kube-api-access-w64gn\") pod \"maas-api-revoke-sub-2135802a-chnnp\" (UID: \"f22c82c7-938a-47d9-bf58-6d204d63794f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:27.374000 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.373968    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:27.676559 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:27.676528    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"]
Nov 02 13:57:27.677101 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:57:27.677076    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22c82c7_938a_47d9_bf58_6d204d63794f.slice/crio-c322aff444c124db43c5e79405a114050286e1ea01c953de27f8e573403333e2 WatchSource:0}: Error finding container c322aff444c124db43c5e79405a114050286e1ea01c953de27f8e573403333e2: Status 404 returned error can't find the container with id c322aff444c124db43c5e79405a114050286e1ea01c953de27f8e573403333e2
Nov 02 13:57:28.476017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:28.475956    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp" event={"ID":"f22c82c7-938a-47d9-bf58-6d204d63794f","Type":"ContainerStarted","Data":"56fc962dd4e8e4e392de2dc50144983dab5afc9b27c61dca4432695c0f733644"}
Nov 02 13:57:28.476417 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:28.476043    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp" event={"ID":"f22c82c7-938a-47d9-bf58-6d204d63794f","Type":"ContainerStarted","Data":"c322aff444c124db43c5e79405a114050286e1ea01c953de27f8e573403333e2"}
Nov 02 13:57:29.482940 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:29.482901    5879 generic.go:356] "Generic (PLEG): container finished" podID="f22c82c7-938a-47d9-bf58-6d204d63794f" containerID="56fc962dd4e8e4e392de2dc50144983dab5afc9b27c61dca4432695c0f733644" exitCode=0
Nov 02 13:57:29.483376 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:29.482983    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp" event={"ID":"f22c82c7-938a-47d9-bf58-6d204d63794f","Type":"ContainerDied","Data":"56fc962dd4e8e4e392de2dc50144983dab5afc9b27c61dca4432695c0f733644"}
Nov 02 13:57:30.627075 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:30.627044    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:30.721443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:30.721404    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-openshift-ca\") pod \"f22c82c7-938a-47d9-bf58-6d204d63794f\" (UID: \"f22c82c7-938a-47d9-bf58-6d204d63794f\") "
Nov 02 13:57:30.721443 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:30.721448    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-xks-ca\") pod \"f22c82c7-938a-47d9-bf58-6d204d63794f\" (UID: \"f22c82c7-938a-47d9-bf58-6d204d63794f\") "
Nov 02 13:57:30.721737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:30.721487    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22c82c7-938a-47d9-bf58-6d204d63794f-kube-api-access-w64gn\" (UniqueName: \"kubernetes.io/projected/f22c82c7-938a-47d9-bf58-6d204d63794f-kube-api-access-w64gn\") pod \"f22c82c7-938a-47d9-bf58-6d204d63794f\" (UID: \"f22c82c7-938a-47d9-bf58-6d204d63794f\") "
Nov 02 13:57:30.721833 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:30.721795    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-openshift-ca" pod "f22c82c7-938a-47d9-bf58-6d204d63794f" (UID: "f22c82c7-938a-47d9-bf58-6d204d63794f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:57:30.721905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:30.721887    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:57:30.723872 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:30.723847    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-xks-ca" pod "f22c82c7-938a-47d9-bf58-6d204d63794f" (UID: "f22c82c7-938a-47d9-bf58-6d204d63794f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:57:30.723971 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:30.723871    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22c82c7-938a-47d9-bf58-6d204d63794f-kube-api-access-w64gn" pod "f22c82c7-938a-47d9-bf58-6d204d63794f" (UID: "f22c82c7-938a-47d9-bf58-6d204d63794f"). InnerVolumeSpecName "kube-api-access-w64gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:57:30.822873 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:30.822791    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w64gn\" (UniqueName: \"kubernetes.io/projected/f22c82c7-938a-47d9-bf58-6d204d63794f-kube-api-access-w64gn\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:57:30.822873 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:30.822822    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22c82c7-938a-47d9-bf58-6d204d63794f-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:57:31.494280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:31.494242    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp" event={"ID":"f22c82c7-938a-47d9-bf58-6d204d63794f","Type":"ContainerDied","Data":"c322aff444c124db43c5e79405a114050286e1ea01c953de27f8e573403333e2"}
Nov 02 13:57:31.494280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:31.494279    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"
Nov 02 13:57:31.494521 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:31.494291    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c322aff444c124db43c5e79405a114050286e1ea01c953de27f8e573403333e2"
Nov 02 13:57:44.875148 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:57:44.875063    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:57:44.963421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:44.963382    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:45.860177 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:45.860121    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:57:45.860440 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:45.860216    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:57:52.118804 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:52.118761    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:57:59.738670 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.738628    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"]
Nov 02 13:57:59.739279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.739211    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f22c82c7-938a-47d9-bf58-6d204d63794f" containerName="revoke-keys"
Nov 02 13:57:59.739279 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.739229    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22c82c7-938a-47d9-bf58-6d204d63794f" containerName="revoke-keys"
Nov 02 13:57:59.739385 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.739331    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22c82c7-938a-47d9-bf58-6d204d63794f" containerName="revoke-keys"
Nov 02 13:57:59.742668 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.742651    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:57:59.773758 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.773728    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:57:59.774080 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.774054    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:57:59.775887 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.775863    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"]
Nov 02 13:57:59.854074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.854034    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85d5088e-5ljj5\" (UID: \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:57:59.854074 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.854078    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85d5088e-5ljj5\" (UID: \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:57:59.854323 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.854299    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bn2p\" (UniqueName: \"kubernetes.io/projected/399ee53f-cec5-4d50-80b7-afaa8e3aff94-kube-api-access-2bn2p\") pod \"maas-api-revoke-sub-85d5088e-5ljj5\" (UID: \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:57:59.955737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.955692    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bn2p\" (UniqueName: \"kubernetes.io/projected/399ee53f-cec5-4d50-80b7-afaa8e3aff94-kube-api-access-2bn2p\") pod \"maas-api-revoke-sub-85d5088e-5ljj5\" (UID: \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:57:59.955944 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.955789    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85d5088e-5ljj5\" (UID: \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:57:59.955944 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.955817    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85d5088e-5ljj5\" (UID: \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:57:59.956567 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.956539    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85d5088e-5ljj5\" (UID: \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:57:59.958419 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.958400    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85d5088e-5ljj5\" (UID: \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:57:59.981562 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:57:59.981530    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bn2p\" (UniqueName: \"kubernetes.io/projected/399ee53f-cec5-4d50-80b7-afaa8e3aff94-kube-api-access-2bn2p\") pod \"maas-api-revoke-sub-85d5088e-5ljj5\" (UID: \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:58:00.060421 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:00.060328    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:58:00.363408 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:00.363380    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"]
Nov 02 13:58:00.364029 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:58:00.363978    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399ee53f_cec5_4d50_80b7_afaa8e3aff94.slice/crio-ae6e663979ead4dfef43bfde46d935c0d6421568c0945bea341fe3c50f9fe5ff WatchSource:0}: Error finding container ae6e663979ead4dfef43bfde46d935c0d6421568c0945bea341fe3c50f9fe5ff: Status 404 returned error can't find the container with id ae6e663979ead4dfef43bfde46d935c0d6421568c0945bea341fe3c50f9fe5ff
Nov 02 13:58:00.666881 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:00.666844    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5" event={"ID":"399ee53f-cec5-4d50-80b7-afaa8e3aff94","Type":"ContainerStarted","Data":"ae6e663979ead4dfef43bfde46d935c0d6421568c0945bea341fe3c50f9fe5ff"}
Nov 02 13:58:01.673428 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:01.673381    5879 generic.go:356] "Generic (PLEG): container finished" podID="399ee53f-cec5-4d50-80b7-afaa8e3aff94" containerID="1481b738a7813dc510ec8bbfde4b68e9c53ac4afc97d74d2dd269aaca560f7d9" exitCode=0
Nov 02 13:58:01.673858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:01.673459    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5" event={"ID":"399ee53f-cec5-4d50-80b7-afaa8e3aff94","Type":"ContainerDied","Data":"1481b738a7813dc510ec8bbfde4b68e9c53ac4afc97d74d2dd269aaca560f7d9"}
Nov 02 13:58:02.819693 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:02.819664    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:58:02.985910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:02.985810    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-xks-ca\") pod \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\" (UID: \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\") "
Nov 02 13:58:02.985910 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:02.985892    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/399ee53f-cec5-4d50-80b7-afaa8e3aff94-kube-api-access-2bn2p\" (UniqueName: \"kubernetes.io/projected/399ee53f-cec5-4d50-80b7-afaa8e3aff94-kube-api-access-2bn2p\") pod \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\" (UID: \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\") "
Nov 02 13:58:02.986168 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:02.986047    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-openshift-ca\") pod \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\" (UID: \"399ee53f-cec5-4d50-80b7-afaa8e3aff94\") "
Nov 02 13:58:02.986438 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:02.986406    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-openshift-ca" pod "399ee53f-cec5-4d50-80b7-afaa8e3aff94" (UID: "399ee53f-cec5-4d50-80b7-afaa8e3aff94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:02.988267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:02.988241    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399ee53f-cec5-4d50-80b7-afaa8e3aff94-kube-api-access-2bn2p" pod "399ee53f-cec5-4d50-80b7-afaa8e3aff94" (UID: "399ee53f-cec5-4d50-80b7-afaa8e3aff94"). InnerVolumeSpecName "kube-api-access-2bn2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:02.988267 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:02.988259    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-xks-ca" pod "399ee53f-cec5-4d50-80b7-afaa8e3aff94" (UID: "399ee53f-cec5-4d50-80b7-afaa8e3aff94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:03.087825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:03.087788    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:58:03.087825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:03.087819    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bn2p\" (UniqueName: \"kubernetes.io/projected/399ee53f-cec5-4d50-80b7-afaa8e3aff94-kube-api-access-2bn2p\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:58:03.087825 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:03.087829    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/399ee53f-cec5-4d50-80b7-afaa8e3aff94-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:58:03.684623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:03.684586    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5"
Nov 02 13:58:03.684623 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:03.684601    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85d5088e-5ljj5" event={"ID":"399ee53f-cec5-4d50-80b7-afaa8e3aff94","Type":"ContainerDied","Data":"ae6e663979ead4dfef43bfde46d935c0d6421568c0945bea341fe3c50f9fe5ff"}
Nov 02 13:58:03.684864 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:03.684657    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae6e663979ead4dfef43bfde46d935c0d6421568c0945bea341fe3c50f9fe5ff"
Nov 02 13:58:10.094182 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.094146    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"]
Nov 02 13:58:10.094801 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.094780    5879 cpu_manager.go:414] "containerMap: removing container" podUID="399ee53f-cec5-4d50-80b7-afaa8e3aff94" containerName="revoke-keys"
Nov 02 13:58:10.094905 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.094804    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="399ee53f-cec5-4d50-80b7-afaa8e3aff94" containerName="revoke-keys"
Nov 02 13:58:10.094963 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.094909    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="399ee53f-cec5-4d50-80b7-afaa8e3aff94" containerName="revoke-keys"
Nov 02 13:58:10.098392 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.098371    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:10.117055 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.117019    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:58:10.117055 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.117027    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:58:10.133283 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.133251    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"]
Nov 02 13:58:10.162058 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.162013    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5cf680c-psslk\" (UID: \"0dda6167-10a1-4679-821b-fd09ca5eccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:10.162282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.162164    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qtzm\" (UniqueName: \"kubernetes.io/projected/0dda6167-10a1-4679-821b-fd09ca5eccf5-kube-api-access-9qtzm\") pod \"maas-api-revoke-sub-c5cf680c-psslk\" (UID: \"0dda6167-10a1-4679-821b-fd09ca5eccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:10.162282 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.162240    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5cf680c-psslk\" (UID: \"0dda6167-10a1-4679-821b-fd09ca5eccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:10.263379 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.263332    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qtzm\" (UniqueName: \"kubernetes.io/projected/0dda6167-10a1-4679-821b-fd09ca5eccf5-kube-api-access-9qtzm\") pod \"maas-api-revoke-sub-c5cf680c-psslk\" (UID: \"0dda6167-10a1-4679-821b-fd09ca5eccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:10.263608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.263413    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5cf680c-psslk\" (UID: \"0dda6167-10a1-4679-821b-fd09ca5eccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:10.263608 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.263488    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5cf680c-psslk\" (UID: \"0dda6167-10a1-4679-821b-fd09ca5eccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:10.264252 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.264227    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5cf680c-psslk\" (UID: \"0dda6167-10a1-4679-821b-fd09ca5eccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:10.266082 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.266060    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5cf680c-psslk\" (UID: \"0dda6167-10a1-4679-821b-fd09ca5eccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:10.289738 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.289711    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qtzm\" (UniqueName: \"kubernetes.io/projected/0dda6167-10a1-4679-821b-fd09ca5eccf5-kube-api-access-9qtzm\") pod \"maas-api-revoke-sub-c5cf680c-psslk\" (UID: \"0dda6167-10a1-4679-821b-fd09ca5eccf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:10.417649 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.417608    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:10.725377 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:10.725344    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"]
Nov 02 13:58:10.725692 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:58:10.725666    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dda6167_10a1_4679_821b_fd09ca5eccf5.slice/crio-6c88298e537ff628fa52a52dd5d789c345564bcf49be2139fa4a84c0cbd4b2d9 WatchSource:0}: Error finding container 6c88298e537ff628fa52a52dd5d789c345564bcf49be2139fa4a84c0cbd4b2d9: Status 404 returned error can't find the container with id 6c88298e537ff628fa52a52dd5d789c345564bcf49be2139fa4a84c0cbd4b2d9
Nov 02 13:58:11.728119 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:11.728085    5879 generic.go:356] "Generic (PLEG): container finished" podID="0dda6167-10a1-4679-821b-fd09ca5eccf5" containerID="62390762118ce1f7437db167856c7db2a86f6d9cd108e62ca7cab4047912bfb0" exitCode=0
Nov 02 13:58:11.728525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:11.728181    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk" event={"ID":"0dda6167-10a1-4679-821b-fd09ca5eccf5","Type":"ContainerDied","Data":"62390762118ce1f7437db167856c7db2a86f6d9cd108e62ca7cab4047912bfb0"}
Nov 02 13:58:11.728525 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:11.728241    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk" event={"ID":"0dda6167-10a1-4679-821b-fd09ca5eccf5","Type":"ContainerStarted","Data":"6c88298e537ff628fa52a52dd5d789c345564bcf49be2139fa4a84c0cbd4b2d9"}
Nov 02 13:58:12.876054 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:12.876015    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:12.990331 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:12.990231    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-xks-ca\") pod \"0dda6167-10a1-4679-821b-fd09ca5eccf5\" (UID: \"0dda6167-10a1-4679-821b-fd09ca5eccf5\") "
Nov 02 13:58:12.990529 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:12.990354    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-openshift-ca\") pod \"0dda6167-10a1-4679-821b-fd09ca5eccf5\" (UID: \"0dda6167-10a1-4679-821b-fd09ca5eccf5\") "
Nov 02 13:58:12.990529 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:12.990400    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dda6167-10a1-4679-821b-fd09ca5eccf5-kube-api-access-9qtzm\" (UniqueName: \"kubernetes.io/projected/0dda6167-10a1-4679-821b-fd09ca5eccf5-kube-api-access-9qtzm\") pod \"0dda6167-10a1-4679-821b-fd09ca5eccf5\" (UID: \"0dda6167-10a1-4679-821b-fd09ca5eccf5\") "
Nov 02 13:58:12.990719 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:12.990694    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-openshift-ca" pod "0dda6167-10a1-4679-821b-fd09ca5eccf5" (UID: "0dda6167-10a1-4679-821b-fd09ca5eccf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:12.992576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:12.992549    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-xks-ca" pod "0dda6167-10a1-4679-821b-fd09ca5eccf5" (UID: "0dda6167-10a1-4679-821b-fd09ca5eccf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:12.992685 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:12.992612    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dda6167-10a1-4679-821b-fd09ca5eccf5-kube-api-access-9qtzm" pod "0dda6167-10a1-4679-821b-fd09ca5eccf5" (UID: "0dda6167-10a1-4679-821b-fd09ca5eccf5"). InnerVolumeSpecName "kube-api-access-9qtzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:13.091452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:13.091408    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:58:13.091452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:13.091445    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dda6167-10a1-4679-821b-fd09ca5eccf5-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:58:13.091452 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:13.091455    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qtzm\" (UniqueName: \"kubernetes.io/projected/0dda6167-10a1-4679-821b-fd09ca5eccf5-kube-api-access-9qtzm\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:58:13.739181 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:13.739151    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk"
Nov 02 13:58:13.739353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:13.739148    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5cf680c-psslk" event={"ID":"0dda6167-10a1-4679-821b-fd09ca5eccf5","Type":"ContainerDied","Data":"6c88298e537ff628fa52a52dd5d789c345564bcf49be2139fa4a84c0cbd4b2d9"}
Nov 02 13:58:13.739353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:13.739291    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c88298e537ff628fa52a52dd5d789c345564bcf49be2139fa4a84c0cbd4b2d9"
Nov 02 13:58:14.821078 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:58:14.821039    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:58:41.047677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.047631    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"]
Nov 02 13:58:41.048470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.048093    5879 cpu_manager.go:414] "containerMap: removing container" podUID="0dda6167-10a1-4679-821b-fd09ca5eccf5" containerName="revoke-keys"
Nov 02 13:58:41.048470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.048105    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dda6167-10a1-4679-821b-fd09ca5eccf5" containerName="revoke-keys"
Nov 02 13:58:41.048470 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.048196    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dda6167-10a1-4679-821b-fd09ca5eccf5" containerName="revoke-keys"
Nov 02 13:58:41.051569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.051549    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:41.072327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.072293    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:58:41.074255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.074222    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"]
Nov 02 13:58:41.086236 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.086203    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:58:41.117056 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.117023    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbt6r\" (UniqueName: \"kubernetes.io/projected/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-kube-api-access-sbt6r\") pod \"maas-api-revoke-sub-5efb4fe6-jm2zd\" (UID: \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:41.117224 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.117132    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5efb4fe6-jm2zd\" (UID: \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:41.117224 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.117194    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5efb4fe6-jm2zd\" (UID: \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:41.218394 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.218354    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5efb4fe6-jm2zd\" (UID: \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:41.218606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.218419    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5efb4fe6-jm2zd\" (UID: \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:41.218606 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.218519    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbt6r\" (UniqueName: \"kubernetes.io/projected/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-kube-api-access-sbt6r\") pod \"maas-api-revoke-sub-5efb4fe6-jm2zd\" (UID: \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:41.219370 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.219341    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5efb4fe6-jm2zd\" (UID: \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:41.220879 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.220858    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5efb4fe6-jm2zd\" (UID: \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:41.244675 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.244639    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbt6r\" (UniqueName: \"kubernetes.io/projected/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-kube-api-access-sbt6r\") pod \"maas-api-revoke-sub-5efb4fe6-jm2zd\" (UID: \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:41.365585 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.365552    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:41.668027 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:58:41.667973    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb11133c_62d7_4b85_acbb_0c3f7dbdb1d9.slice/crio-601b8c73cbf3912ac3ffe3dacc06200b40e0bb9417e4f04126583022f3a21b03 WatchSource:0}: Error finding container 601b8c73cbf3912ac3ffe3dacc06200b40e0bb9417e4f04126583022f3a21b03: Status 404 returned error can't find the container with id 601b8c73cbf3912ac3ffe3dacc06200b40e0bb9417e4f04126583022f3a21b03
Nov 02 13:58:41.668198 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.668135    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"]
Nov 02 13:58:41.972349 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:41.972261    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd" event={"ID":"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9","Type":"ContainerStarted","Data":"601b8c73cbf3912ac3ffe3dacc06200b40e0bb9417e4f04126583022f3a21b03"}
Nov 02 13:58:42.977984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:42.977947    5879 generic.go:356] "Generic (PLEG): container finished" podID="fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9" containerID="47b7a50dd82a9c2570c57ada9c8f1201159bbb8a8dc229e51f32356f435cf919" exitCode=0
Nov 02 13:58:42.978458 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:42.978031    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd" event={"ID":"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9","Type":"ContainerDied","Data":"47b7a50dd82a9c2570c57ada9c8f1201159bbb8a8dc229e51f32356f435cf919"}
Nov 02 13:58:44.126305 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.126272    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:44.250596 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.250493    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-xks-ca\") pod \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\" (UID: \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\") "
Nov 02 13:58:44.250596 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.250548    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-openshift-ca\") pod \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\" (UID: \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\") "
Nov 02 13:58:44.250838 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.250637    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-kube-api-access-sbt6r\" (UniqueName: \"kubernetes.io/projected/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-kube-api-access-sbt6r\") pod \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\" (UID: \"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9\") "
Nov 02 13:58:44.250965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.250936    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-openshift-ca" pod "fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9" (UID: "fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:44.251076 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.251061    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:58:44.252813 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.252796    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-xks-ca" pod "fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9" (UID: "fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:44.252937 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.252916    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-kube-api-access-sbt6r" pod "fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9" (UID: "fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9"). InnerVolumeSpecName "kube-api-access-sbt6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:44.351754 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.351712    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:58:44.351754 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.351745    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbt6r\" (UniqueName: \"kubernetes.io/projected/fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9-kube-api-access-sbt6r\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:58:44.821664 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:58:44.821626    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:58:44.988866 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.988836    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd"
Nov 02 13:58:44.989068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.988875    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efb4fe6-jm2zd" event={"ID":"fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9","Type":"ContainerDied","Data":"601b8c73cbf3912ac3ffe3dacc06200b40e0bb9417e4f04126583022f3a21b03"}
Nov 02 13:58:44.989068 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:44.988921    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="601b8c73cbf3912ac3ffe3dacc06200b40e0bb9417e4f04126583022f3a21b03"
Nov 02 13:58:45.127280 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:45.127245    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:45.254496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:45.254463    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:49.207087 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:49.207051    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:58:51.398106 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.398059    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"]
Nov 02 13:58:51.399101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.399072    5879 cpu_manager.go:414] "containerMap: removing container" podUID="fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9" containerName="revoke-keys"
Nov 02 13:58:51.399101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.399102    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9" containerName="revoke-keys"
Nov 02 13:58:51.399371 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.399353    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9" containerName="revoke-keys"
Nov 02 13:58:51.403576 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.403545    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:51.416338 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.416304    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"]
Nov 02 13:58:51.427726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.427695    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lw8c\" (UniqueName: \"kubernetes.io/projected/535905be-87fb-4071-a92d-416703c0a626-kube-api-access-2lw8c\") pod \"maas-api-revoke-sub-bf8f1462-tg7gn\" (UID: \"535905be-87fb-4071-a92d-416703c0a626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:51.427922 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.427809    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535905be-87fb-4071-a92d-416703c0a626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf8f1462-tg7gn\" (UID: \"535905be-87fb-4071-a92d-416703c0a626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:51.427922 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.427830    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535905be-87fb-4071-a92d-416703c0a626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf8f1462-tg7gn\" (UID: \"535905be-87fb-4071-a92d-416703c0a626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:51.428427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.428404    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:58:51.428427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.428418    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:58:51.528979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.528930    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535905be-87fb-4071-a92d-416703c0a626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf8f1462-tg7gn\" (UID: \"535905be-87fb-4071-a92d-416703c0a626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:51.528979 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.528981    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535905be-87fb-4071-a92d-416703c0a626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf8f1462-tg7gn\" (UID: \"535905be-87fb-4071-a92d-416703c0a626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:51.529255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.529099    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lw8c\" (UniqueName: \"kubernetes.io/projected/535905be-87fb-4071-a92d-416703c0a626-kube-api-access-2lw8c\") pod \"maas-api-revoke-sub-bf8f1462-tg7gn\" (UID: \"535905be-87fb-4071-a92d-416703c0a626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:51.529796 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.529767    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535905be-87fb-4071-a92d-416703c0a626-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf8f1462-tg7gn\" (UID: \"535905be-87fb-4071-a92d-416703c0a626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:51.531616 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.531588    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535905be-87fb-4071-a92d-416703c0a626-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf8f1462-tg7gn\" (UID: \"535905be-87fb-4071-a92d-416703c0a626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:51.555137 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.555104    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lw8c\" (UniqueName: \"kubernetes.io/projected/535905be-87fb-4071-a92d-416703c0a626-kube-api-access-2lw8c\") pod \"maas-api-revoke-sub-bf8f1462-tg7gn\" (UID: \"535905be-87fb-4071-a92d-416703c0a626\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:51.720256 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:51.720161    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:52.021690 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:52.021655    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"]
Nov 02 13:58:52.022125 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:58:52.022094    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod535905be_87fb_4071_a92d_416703c0a626.slice/crio-a0d891e09059248c5f5959df834c3cbf35bf0f67755588d736d47f634e2b6ede WatchSource:0}: Error finding container a0d891e09059248c5f5959df834c3cbf35bf0f67755588d736d47f634e2b6ede: Status 404 returned error can't find the container with id a0d891e09059248c5f5959df834c3cbf35bf0f67755588d736d47f634e2b6ede
Nov 02 13:58:53.033272 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:53.033232    5879 generic.go:356] "Generic (PLEG): container finished" podID="535905be-87fb-4071-a92d-416703c0a626" containerID="a08fdc1da3221ac39c947cc807e7ac1e16d3fb9152e8feb0dcb8d977081ecd2d" exitCode=0
Nov 02 13:58:53.033683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:53.033329    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn" event={"ID":"535905be-87fb-4071-a92d-416703c0a626","Type":"ContainerDied","Data":"a08fdc1da3221ac39c947cc807e7ac1e16d3fb9152e8feb0dcb8d977081ecd2d"}
Nov 02 13:58:53.033683 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:53.033391    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn" event={"ID":"535905be-87fb-4071-a92d-416703c0a626","Type":"ContainerStarted","Data":"a0d891e09059248c5f5959df834c3cbf35bf0f67755588d736d47f634e2b6ede"}
Nov 02 13:58:54.202827 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:54.202787    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:54.258084 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:54.258045    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/535905be-87fb-4071-a92d-416703c0a626-kube-api-access-2lw8c\" (UniqueName: \"kubernetes.io/projected/535905be-87fb-4071-a92d-416703c0a626-kube-api-access-2lw8c\") pod \"535905be-87fb-4071-a92d-416703c0a626\" (UID: \"535905be-87fb-4071-a92d-416703c0a626\") "
Nov 02 13:58:54.258255 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:54.258221    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/535905be-87fb-4071-a92d-416703c0a626-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535905be-87fb-4071-a92d-416703c0a626-maas-api-xks-ca\") pod \"535905be-87fb-4071-a92d-416703c0a626\" (UID: \"535905be-87fb-4071-a92d-416703c0a626\") "
Nov 02 13:58:54.258353 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:54.258335    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/535905be-87fb-4071-a92d-416703c0a626-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535905be-87fb-4071-a92d-416703c0a626-maas-api-openshift-ca\") pod \"535905be-87fb-4071-a92d-416703c0a626\" (UID: \"535905be-87fb-4071-a92d-416703c0a626\") "
Nov 02 13:58:54.259402 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:54.259082    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/535905be-87fb-4071-a92d-416703c0a626-maas-api-openshift-ca" pod "535905be-87fb-4071-a92d-416703c0a626" (UID: "535905be-87fb-4071-a92d-416703c0a626"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:58:54.261445 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:54.261417    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/535905be-87fb-4071-a92d-416703c0a626-maas-api-xks-ca" pod "535905be-87fb-4071-a92d-416703c0a626" (UID: "535905be-87fb-4071-a92d-416703c0a626"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:58:54.261772 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:54.261753    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/535905be-87fb-4071-a92d-416703c0a626-kube-api-access-2lw8c" pod "535905be-87fb-4071-a92d-416703c0a626" (UID: "535905be-87fb-4071-a92d-416703c0a626"). InnerVolumeSpecName "kube-api-access-2lw8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:58:54.359888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:54.359789    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/535905be-87fb-4071-a92d-416703c0a626-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:58:54.359888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:54.359831    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lw8c\" (UniqueName: \"kubernetes.io/projected/535905be-87fb-4071-a92d-416703c0a626-kube-api-access-2lw8c\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:58:54.359888 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:54.359847    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/535905be-87fb-4071-a92d-416703c0a626-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:58:55.057028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:55.053726    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn" event={"ID":"535905be-87fb-4071-a92d-416703c0a626","Type":"ContainerDied","Data":"a0d891e09059248c5f5959df834c3cbf35bf0f67755588d736d47f634e2b6ede"}
Nov 02 13:58:55.057028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:55.053798    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0d891e09059248c5f5959df834c3cbf35bf0f67755588d736d47f634e2b6ede"
Nov 02 13:58:55.057028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:55.053877    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf8f1462-tg7gn"
Nov 02 13:58:56.014019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:56.012086    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9tbj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 13:58:56.014019 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:58:56.012152    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" podUID="3a8de801-f2f2-4b08-ad24-022a2e063bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 13:59:14.819715 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:59:14.819627    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:59:23.930964 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:23.930929    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"]
Nov 02 13:59:23.931413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:23.931356    5879 cpu_manager.go:414] "containerMap: removing container" podUID="535905be-87fb-4071-a92d-416703c0a626" containerName="revoke-keys"
Nov 02 13:59:23.931413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:23.931368    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="535905be-87fb-4071-a92d-416703c0a626" containerName="revoke-keys"
Nov 02 13:59:23.931511 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:23.931460    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="535905be-87fb-4071-a92d-416703c0a626" containerName="revoke-keys"
Nov 02 13:59:23.934569 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:23.934554    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:23.953287 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:23.953258    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:23.967325 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:23.967291    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:59:23.967719 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:23.967693    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fefb5b75-j5c6v\" (UID: \"f3f04f59-c343-4387-944a-6caf51ef51e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:23.967830 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:23.967737    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fefb5b75-j5c6v\" (UID: \"f3f04f59-c343-4387-944a-6caf51ef51e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:23.967960 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:23.967944    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f475s\" (UniqueName: \"kubernetes.io/projected/f3f04f59-c343-4387-944a-6caf51ef51e2-kube-api-access-f475s\") pod \"maas-api-revoke-sub-fefb5b75-j5c6v\" (UID: \"f3f04f59-c343-4387-944a-6caf51ef51e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:23.969312 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:23.969292    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"]
Nov 02 13:59:24.069349 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:24.069306    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fefb5b75-j5c6v\" (UID: \"f3f04f59-c343-4387-944a-6caf51ef51e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:24.069561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:24.069358    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fefb5b75-j5c6v\" (UID: \"f3f04f59-c343-4387-944a-6caf51ef51e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:24.069561 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:24.069519    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f475s\" (UniqueName: \"kubernetes.io/projected/f3f04f59-c343-4387-944a-6caf51ef51e2-kube-api-access-f475s\") pod \"maas-api-revoke-sub-fefb5b75-j5c6v\" (UID: \"f3f04f59-c343-4387-944a-6caf51ef51e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:24.070072 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:24.070047    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fefb5b75-j5c6v\" (UID: \"f3f04f59-c343-4387-944a-6caf51ef51e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:24.071843 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:24.071823    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fefb5b75-j5c6v\" (UID: \"f3f04f59-c343-4387-944a-6caf51ef51e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:24.095222 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:24.095195    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f475s\" (UniqueName: \"kubernetes.io/projected/f3f04f59-c343-4387-944a-6caf51ef51e2-kube-api-access-f475s\") pod \"maas-api-revoke-sub-fefb5b75-j5c6v\" (UID: \"f3f04f59-c343-4387-944a-6caf51ef51e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:24.246877 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:24.246796    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:24.550684 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:24.550657    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"]
Nov 02 13:59:24.551080 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:59:24.551054    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f04f59_c343_4387_944a_6caf51ef51e2.slice/crio-c985d6ce599546637b2bd756b86b5c06a0248a909a3cfcc87f03866fe6667677 WatchSource:0}: Error finding container c985d6ce599546637b2bd756b86b5c06a0248a909a3cfcc87f03866fe6667677: Status 404 returned error can't find the container with id c985d6ce599546637b2bd756b86b5c06a0248a909a3cfcc87f03866fe6667677
Nov 02 13:59:25.263155 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:25.263118    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v" event={"ID":"f3f04f59-c343-4387-944a-6caf51ef51e2","Type":"ContainerStarted","Data":"51e03179f0d3fbf30d47558733fd5dba6eba2fdcc3e51dff6ae15c9f7b3a349b"}
Nov 02 13:59:25.263527 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:25.263171    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v" event={"ID":"f3f04f59-c343-4387-944a-6caf51ef51e2","Type":"ContainerStarted","Data":"c985d6ce599546637b2bd756b86b5c06a0248a909a3cfcc87f03866fe6667677"}
Nov 02 13:59:25.307680 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:25.307629    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v" podStartSLOduration=2.307612213 podStartE2EDuration="2.307612213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:25.307169221 +0000 UTC m=+5511.413872073" watchObservedRunningTime="2026-11-02 13:59:25.307612213 +0000 UTC m=+5511.414315065"
Nov 02 13:59:26.269102 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:26.269067    5879 generic.go:356] "Generic (PLEG): container finished" podID="f3f04f59-c343-4387-944a-6caf51ef51e2" containerID="51e03179f0d3fbf30d47558733fd5dba6eba2fdcc3e51dff6ae15c9f7b3a349b" exitCode=0
Nov 02 13:59:26.269579 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:26.269122    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v" event={"ID":"f3f04f59-c343-4387-944a-6caf51ef51e2","Type":"ContainerDied","Data":"51e03179f0d3fbf30d47558733fd5dba6eba2fdcc3e51dff6ae15c9f7b3a349b"}
Nov 02 13:59:27.411276 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:27.411248    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:27.503919 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:27.503876    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-openshift-ca\") pod \"f3f04f59-c343-4387-944a-6caf51ef51e2\" (UID: \"f3f04f59-c343-4387-944a-6caf51ef51e2\") "
Nov 02 13:59:27.504152 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:27.503964    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f04f59-c343-4387-944a-6caf51ef51e2-kube-api-access-f475s\" (UniqueName: \"kubernetes.io/projected/f3f04f59-c343-4387-944a-6caf51ef51e2-kube-api-access-f475s\") pod \"f3f04f59-c343-4387-944a-6caf51ef51e2\" (UID: \"f3f04f59-c343-4387-944a-6caf51ef51e2\") "
Nov 02 13:59:27.504152 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:27.504116    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-xks-ca\") pod \"f3f04f59-c343-4387-944a-6caf51ef51e2\" (UID: \"f3f04f59-c343-4387-944a-6caf51ef51e2\") "
Nov 02 13:59:27.504315 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:27.504293    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-openshift-ca" pod "f3f04f59-c343-4387-944a-6caf51ef51e2" (UID: "f3f04f59-c343-4387-944a-6caf51ef51e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:27.504413 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:27.504399    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:59:27.506329 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:27.506305    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f04f59-c343-4387-944a-6caf51ef51e2-kube-api-access-f475s" pod "f3f04f59-c343-4387-944a-6caf51ef51e2" (UID: "f3f04f59-c343-4387-944a-6caf51ef51e2"). InnerVolumeSpecName "kube-api-access-f475s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:27.506414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:27.506334    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-xks-ca" pod "f3f04f59-c343-4387-944a-6caf51ef51e2" (UID: "f3f04f59-c343-4387-944a-6caf51ef51e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:27.605966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:27.605886    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f04f59-c343-4387-944a-6caf51ef51e2-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:59:27.605966 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:27.605916    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f475s\" (UniqueName: \"kubernetes.io/projected/f3f04f59-c343-4387-944a-6caf51ef51e2-kube-api-access-f475s\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:59:28.280938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:28.280855    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v"
Nov 02 13:59:28.280938 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:28.280898    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fefb5b75-j5c6v" event={"ID":"f3f04f59-c343-4387-944a-6caf51ef51e2","Type":"ContainerDied","Data":"c985d6ce599546637b2bd756b86b5c06a0248a909a3cfcc87f03866fe6667677"}
Nov 02 13:59:28.281162 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:28.280947    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c985d6ce599546637b2bd756b86b5c06a0248a909a3cfcc87f03866fe6667677"
Nov 02 13:59:34.371726 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.371693    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"]
Nov 02 13:59:34.372154 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.372143    5879 cpu_manager.go:414] "containerMap: removing container" podUID="f3f04f59-c343-4387-944a-6caf51ef51e2" containerName="revoke-keys"
Nov 02 13:59:34.372218 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.372155    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f04f59-c343-4387-944a-6caf51ef51e2" containerName="revoke-keys"
Nov 02 13:59:34.372274 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.372245    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f04f59-c343-4387-944a-6caf51ef51e2" containerName="revoke-keys"
Nov 02 13:59:34.375639 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.375618    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:34.392327 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.392302    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:34.406158 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.406131    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:59:34.408342 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.408315    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"]
Nov 02 13:59:34.475414 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.475384    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nrs8\" (UniqueName: \"kubernetes.io/projected/b696b31d-74b1-4e13-a1e7-70d947a62116-kube-api-access-2nrs8\") pod \"maas-api-revoke-sub-5c3e49de-wht8g\" (UID: \"b696b31d-74b1-4e13-a1e7-70d947a62116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:34.475593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.475498    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c3e49de-wht8g\" (UID: \"b696b31d-74b1-4e13-a1e7-70d947a62116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:34.475593 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.475525    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c3e49de-wht8g\" (UID: \"b696b31d-74b1-4e13-a1e7-70d947a62116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:34.577050 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.577012    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nrs8\" (UniqueName: \"kubernetes.io/projected/b696b31d-74b1-4e13-a1e7-70d947a62116-kube-api-access-2nrs8\") pod \"maas-api-revoke-sub-5c3e49de-wht8g\" (UID: \"b696b31d-74b1-4e13-a1e7-70d947a62116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:34.577275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.577084    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c3e49de-wht8g\" (UID: \"b696b31d-74b1-4e13-a1e7-70d947a62116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:34.577275 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.577106    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c3e49de-wht8g\" (UID: \"b696b31d-74b1-4e13-a1e7-70d947a62116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:34.577778 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.577747    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c3e49de-wht8g\" (UID: \"b696b31d-74b1-4e13-a1e7-70d947a62116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:34.579564 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.579544    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c3e49de-wht8g\" (UID: \"b696b31d-74b1-4e13-a1e7-70d947a62116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:34.602564 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.602538    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nrs8\" (UniqueName: \"kubernetes.io/projected/b696b31d-74b1-4e13-a1e7-70d947a62116-kube-api-access-2nrs8\") pod \"maas-api-revoke-sub-5c3e49de-wht8g\" (UID: \"b696b31d-74b1-4e13-a1e7-70d947a62116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:34.689146 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.689064    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:34.988465 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:34.988437    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"]
Nov 02 13:59:34.988797 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:59:34.988774    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb696b31d_74b1_4e13_a1e7_70d947a62116.slice/crio-a3f11bd4e74f01aa9ef8aec20a3afbe5d060f7cafedd6a3a4463ac1c2eb11192 WatchSource:0}: Error finding container a3f11bd4e74f01aa9ef8aec20a3afbe5d060f7cafedd6a3a4463ac1c2eb11192: Status 404 returned error can't find the container with id a3f11bd4e74f01aa9ef8aec20a3afbe5d060f7cafedd6a3a4463ac1c2eb11192
Nov 02 13:59:35.318205 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:35.318177    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g" event={"ID":"b696b31d-74b1-4e13-a1e7-70d947a62116","Type":"ContainerStarted","Data":"38d892edb0190ce2819d7110cf35a513e3d1b839a72be11d021709e3cf9577b0"}
Nov 02 13:59:35.318349 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:35.318260    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g" event={"ID":"b696b31d-74b1-4e13-a1e7-70d947a62116","Type":"ContainerStarted","Data":"a3f11bd4e74f01aa9ef8aec20a3afbe5d060f7cafedd6a3a4463ac1c2eb11192"}
Nov 02 13:59:35.362609 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:35.362556    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g" podStartSLOduration=1.36254212 podStartE2EDuration="1.36254212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:35.361256788 +0000 UTC m=+5521.467959639" watchObservedRunningTime="2026-11-02 13:59:35.36254212 +0000 UTC m=+5521.469244971"
Nov 02 13:59:36.324679 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:36.324582    5879 generic.go:356] "Generic (PLEG): container finished" podID="b696b31d-74b1-4e13-a1e7-70d947a62116" containerID="38d892edb0190ce2819d7110cf35a513e3d1b839a72be11d021709e3cf9577b0" exitCode=0
Nov 02 13:59:36.325110 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:36.324671    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g" event={"ID":"b696b31d-74b1-4e13-a1e7-70d947a62116","Type":"ContainerDied","Data":"38d892edb0190ce2819d7110cf35a513e3d1b839a72be11d021709e3cf9577b0"}
Nov 02 13:59:37.469965 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:37.469938    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:37.608858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:37.608783    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-openshift-ca\") pod \"b696b31d-74b1-4e13-a1e7-70d947a62116\" (UID: \"b696b31d-74b1-4e13-a1e7-70d947a62116\") "
Nov 02 13:59:37.608858 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:37.608855    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b696b31d-74b1-4e13-a1e7-70d947a62116-kube-api-access-2nrs8\" (UniqueName: \"kubernetes.io/projected/b696b31d-74b1-4e13-a1e7-70d947a62116-kube-api-access-2nrs8\") pod \"b696b31d-74b1-4e13-a1e7-70d947a62116\" (UID: \"b696b31d-74b1-4e13-a1e7-70d947a62116\") "
Nov 02 13:59:37.609120 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:37.608891    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-xks-ca\") pod \"b696b31d-74b1-4e13-a1e7-70d947a62116\" (UID: \"b696b31d-74b1-4e13-a1e7-70d947a62116\") "
Nov 02 13:59:37.609200 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:37.609152    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-openshift-ca" pod "b696b31d-74b1-4e13-a1e7-70d947a62116" (UID: "b696b31d-74b1-4e13-a1e7-70d947a62116"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:37.609344 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:37.609316    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:59:37.611211 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:37.611168    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b696b31d-74b1-4e13-a1e7-70d947a62116-kube-api-access-2nrs8" pod "b696b31d-74b1-4e13-a1e7-70d947a62116" (UID: "b696b31d-74b1-4e13-a1e7-70d947a62116"). InnerVolumeSpecName "kube-api-access-2nrs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:37.611313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:37.611215    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-xks-ca" pod "b696b31d-74b1-4e13-a1e7-70d947a62116" (UID: "b696b31d-74b1-4e13-a1e7-70d947a62116"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:37.710984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:37.710951    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nrs8\" (UniqueName: \"kubernetes.io/projected/b696b31d-74b1-4e13-a1e7-70d947a62116-kube-api-access-2nrs8\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:59:37.710984 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:37.710983    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b696b31d-74b1-4e13-a1e7-70d947a62116-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:59:38.339184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:38.339076    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g" event={"ID":"b696b31d-74b1-4e13-a1e7-70d947a62116","Type":"ContainerDied","Data":"a3f11bd4e74f01aa9ef8aec20a3afbe5d060f7cafedd6a3a4463ac1c2eb11192"}
Nov 02 13:59:38.339184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:38.339140    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3f11bd4e74f01aa9ef8aec20a3afbe5d060f7cafedd6a3a4463ac1c2eb11192"
Nov 02 13:59:38.339184 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:38.339122    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c3e49de-wht8g"
Nov 02 13:59:44.696007 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.695950    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"]
Nov 02 13:59:44.696522 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.696504    5879 cpu_manager.go:414] "containerMap: removing container" podUID="b696b31d-74b1-4e13-a1e7-70d947a62116" containerName="revoke-keys"
Nov 02 13:59:44.696522 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.696523    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b696b31d-74b1-4e13-a1e7-70d947a62116" containerName="revoke-keys"
Nov 02 13:59:44.696622 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.696596    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="b696b31d-74b1-4e13-a1e7-70d947a62116" containerName="revoke-keys"
Nov 02 13:59:44.699985 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.699964    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:44.720214 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.720181    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"]
Nov 02 13:59:44.731208 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.731174    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:44.731384 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.731345    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:59:44.803642 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.803606    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93219489-287gv\" (UID: \"dffdf00f-b452-499b-b760-ddeedf3d3aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:44.803811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.803741    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93219489-287gv\" (UID: \"dffdf00f-b452-499b-b760-ddeedf3d3aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:44.803811 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.803797    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q494f\" (UniqueName: \"kubernetes.io/projected/dffdf00f-b452-499b-b760-ddeedf3d3aad-kube-api-access-q494f\") pod \"maas-api-revoke-sub-93219489-287gv\" (UID: \"dffdf00f-b452-499b-b760-ddeedf3d3aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:44.819963 ip-10-0-134-229 kubenswrapper[5879]: E1102 13:59:44.819931    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 13:59:44.904742 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.904708    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93219489-287gv\" (UID: \"dffdf00f-b452-499b-b760-ddeedf3d3aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:44.904960 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.904777    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q494f\" (UniqueName: \"kubernetes.io/projected/dffdf00f-b452-499b-b760-ddeedf3d3aad-kube-api-access-q494f\") pod \"maas-api-revoke-sub-93219489-287gv\" (UID: \"dffdf00f-b452-499b-b760-ddeedf3d3aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:44.904960 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.904874    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93219489-287gv\" (UID: \"dffdf00f-b452-499b-b760-ddeedf3d3aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:44.905427 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.905391    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93219489-287gv\" (UID: \"dffdf00f-b452-499b-b760-ddeedf3d3aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:44.907339 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.907305    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93219489-287gv\" (UID: \"dffdf00f-b452-499b-b760-ddeedf3d3aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:44.930534 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:44.930500    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q494f\" (UniqueName: \"kubernetes.io/projected/dffdf00f-b452-499b-b760-ddeedf3d3aad-kube-api-access-q494f\") pod \"maas-api-revoke-sub-93219489-287gv\" (UID: \"dffdf00f-b452-499b-b760-ddeedf3d3aad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:45.013456 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:45.013373    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:45.223033 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:45.222986    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:45.313607 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:45.313579    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"]
Nov 02 13:59:45.314276 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:59:45.314244    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffdf00f_b452_499b_b760_ddeedf3d3aad.slice/crio-7f422f479bbf70dd2da54a5c92d24bf07fa5f5ac366ff89aed0073b7dcfbeefb WatchSource:0}: Error finding container 7f422f479bbf70dd2da54a5c92d24bf07fa5f5ac366ff89aed0073b7dcfbeefb: Status 404 returned error can't find the container with id 7f422f479bbf70dd2da54a5c92d24bf07fa5f5ac366ff89aed0073b7dcfbeefb
Nov 02 13:59:45.389765 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:45.389734    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv" event={"ID":"dffdf00f-b452-499b-b760-ddeedf3d3aad","Type":"ContainerStarted","Data":"7f422f479bbf70dd2da54a5c92d24bf07fa5f5ac366ff89aed0073b7dcfbeefb"}
Nov 02 13:59:46.395920 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:46.395884    5879 generic.go:356] "Generic (PLEG): container finished" podID="dffdf00f-b452-499b-b760-ddeedf3d3aad" containerID="36dfaf07e2d03d025d186a682873857c64b1a2af3e41840bd97b14c04431192e" exitCode=0
Nov 02 13:59:46.396400 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:46.395966    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv" event={"ID":"dffdf00f-b452-499b-b760-ddeedf3d3aad","Type":"ContainerDied","Data":"36dfaf07e2d03d025d186a682873857c64b1a2af3e41840bd97b14c04431192e"}
Nov 02 13:59:47.545472 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:47.545439    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:47.634017 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:47.633950    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-xks-ca\") pod \"dffdf00f-b452-499b-b760-ddeedf3d3aad\" (UID: \"dffdf00f-b452-499b-b760-ddeedf3d3aad\") "
Nov 02 13:59:47.634201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:47.634101    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-openshift-ca\") pod \"dffdf00f-b452-499b-b760-ddeedf3d3aad\" (UID: \"dffdf00f-b452-499b-b760-ddeedf3d3aad\") "
Nov 02 13:59:47.634201 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:47.634148    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffdf00f-b452-499b-b760-ddeedf3d3aad-kube-api-access-q494f\" (UniqueName: \"kubernetes.io/projected/dffdf00f-b452-499b-b760-ddeedf3d3aad-kube-api-access-q494f\") pod \"dffdf00f-b452-499b-b760-ddeedf3d3aad\" (UID: \"dffdf00f-b452-499b-b760-ddeedf3d3aad\") "
Nov 02 13:59:47.634489 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:47.634458    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-openshift-ca" pod "dffdf00f-b452-499b-b760-ddeedf3d3aad" (UID: "dffdf00f-b452-499b-b760-ddeedf3d3aad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:47.636144 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:47.636116    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-xks-ca" pod "dffdf00f-b452-499b-b760-ddeedf3d3aad" (UID: "dffdf00f-b452-499b-b760-ddeedf3d3aad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:47.636309 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:47.636292    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffdf00f-b452-499b-b760-ddeedf3d3aad-kube-api-access-q494f" pod "dffdf00f-b452-499b-b760-ddeedf3d3aad" (UID: "dffdf00f-b452-499b-b760-ddeedf3d3aad"). InnerVolumeSpecName "kube-api-access-q494f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:47.735748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:47.735651    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:59:47.735748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:47.735697    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffdf00f-b452-499b-b760-ddeedf3d3aad-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:59:47.735748 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:47.735708    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q494f\" (UniqueName: \"kubernetes.io/projected/dffdf00f-b452-499b-b760-ddeedf3d3aad-kube-api-access-q494f\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:59:48.239028 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:48.238972    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:48.357081 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:48.356984    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 13:59:48.407314 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:48.407279    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv" event={"ID":"dffdf00f-b452-499b-b760-ddeedf3d3aad","Type":"ContainerDied","Data":"7f422f479bbf70dd2da54a5c92d24bf07fa5f5ac366ff89aed0073b7dcfbeefb"}
Nov 02 13:59:48.407485 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:48.407331    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f422f479bbf70dd2da54a5c92d24bf07fa5f5ac366ff89aed0073b7dcfbeefb"
Nov 02 13:59:48.407485 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:48.407292    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93219489-287gv"
Nov 02 13:59:55.130251 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.130211    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"]
Nov 02 13:59:55.130760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.130737    5879 cpu_manager.go:414] "containerMap: removing container" podUID="dffdf00f-b452-499b-b760-ddeedf3d3aad" containerName="revoke-keys"
Nov 02 13:59:55.130760 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.130750    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffdf00f-b452-499b-b760-ddeedf3d3aad" containerName="revoke-keys"
Nov 02 13:59:55.130889 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.130839    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffdf00f-b452-499b-b760-ddeedf3d3aad" containerName="revoke-keys"
Nov 02 13:59:55.134254 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.134234    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:55.165716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.165676    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 13:59:55.165716 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.165738    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 13:59:55.167915 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.167887    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"]
Nov 02 13:59:55.317237 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.317192    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2zkv\" (UniqueName: \"kubernetes.io/projected/222e7c04-54d7-4429-9bf2-3943f5e03922-kube-api-access-b2zkv\") pod \"maas-api-revoke-sub-6005186d-mjkxs\" (UID: \"222e7c04-54d7-4429-9bf2-3943f5e03922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:55.317439 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.317295    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6005186d-mjkxs\" (UID: \"222e7c04-54d7-4429-9bf2-3943f5e03922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:55.317496 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.317471    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6005186d-mjkxs\" (UID: \"222e7c04-54d7-4429-9bf2-3943f5e03922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:55.418621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.418525    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2zkv\" (UniqueName: \"kubernetes.io/projected/222e7c04-54d7-4429-9bf2-3943f5e03922-kube-api-access-b2zkv\") pod \"maas-api-revoke-sub-6005186d-mjkxs\" (UID: \"222e7c04-54d7-4429-9bf2-3943f5e03922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:55.418621 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.418569    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6005186d-mjkxs\" (UID: \"222e7c04-54d7-4429-9bf2-3943f5e03922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:55.418884 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.418643    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6005186d-mjkxs\" (UID: \"222e7c04-54d7-4429-9bf2-3943f5e03922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:55.419374 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.419345    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6005186d-mjkxs\" (UID: \"222e7c04-54d7-4429-9bf2-3943f5e03922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:55.421134 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.421113    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6005186d-mjkxs\" (UID: \"222e7c04-54d7-4429-9bf2-3943f5e03922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:55.443969 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.443933    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2zkv\" (UniqueName: \"kubernetes.io/projected/222e7c04-54d7-4429-9bf2-3943f5e03922-kube-api-access-b2zkv\") pod \"maas-api-revoke-sub-6005186d-mjkxs\" (UID: \"222e7c04-54d7-4429-9bf2-3943f5e03922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:55.447800 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.447777    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:55.751677 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:55.751644    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"]
Nov 02 13:59:55.752459 ip-10-0-134-229 kubenswrapper[5879]: W1102 13:59:55.752425    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222e7c04_54d7_4429_9bf2_3943f5e03922.slice/crio-b616b29496f6e994c519baf647bf798063572d0ea98324697c18fef16db02929 WatchSource:0}: Error finding container b616b29496f6e994c519baf647bf798063572d0ea98324697c18fef16db02929: Status 404 returned error can't find the container with id b616b29496f6e994c519baf647bf798063572d0ea98324697c18fef16db02929
Nov 02 13:59:56.449698 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:56.449651    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs" event={"ID":"222e7c04-54d7-4429-9bf2-3943f5e03922","Type":"ContainerStarted","Data":"ab9e7ef63423b4eadd88a808ec2110508e52694a851206721cc06158e6bc2485"}
Nov 02 13:59:56.450147 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:56.449721    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs" event={"ID":"222e7c04-54d7-4429-9bf2-3943f5e03922","Type":"ContainerStarted","Data":"b616b29496f6e994c519baf647bf798063572d0ea98324697c18fef16db02929"}
Nov 02 13:59:56.496258 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:56.496203    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs" podStartSLOduration=1.496178451 podStartE2EDuration="1.496178451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 13:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 13:59:56.495279012 +0000 UTC m=+5542.601981863" watchObservedRunningTime="2026-11-02 13:59:56.496178451 +0000 UTC m=+5542.602881297"
Nov 02 13:59:57.467737 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:57.467687    5879 generic.go:356] "Generic (PLEG): container finished" podID="222e7c04-54d7-4429-9bf2-3943f5e03922" containerID="ab9e7ef63423b4eadd88a808ec2110508e52694a851206721cc06158e6bc2485" exitCode=0
Nov 02 13:59:57.468321 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:57.467760    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs" event={"ID":"222e7c04-54d7-4429-9bf2-3943f5e03922","Type":"ContainerDied","Data":"ab9e7ef63423b4eadd88a808ec2110508e52694a851206721cc06158e6bc2485"}
Nov 02 13:59:58.877313 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:58.877281    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:58.969493 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:58.969456    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-xks-ca\") pod \"222e7c04-54d7-4429-9bf2-3943f5e03922\" (UID: \"222e7c04-54d7-4429-9bf2-3943f5e03922\") "
Nov 02 13:59:58.969688 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:58.969534    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-openshift-ca\") pod \"222e7c04-54d7-4429-9bf2-3943f5e03922\" (UID: \"222e7c04-54d7-4429-9bf2-3943f5e03922\") "
Nov 02 13:59:58.969774 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:58.969691    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222e7c04-54d7-4429-9bf2-3943f5e03922-kube-api-access-b2zkv\" (UniqueName: \"kubernetes.io/projected/222e7c04-54d7-4429-9bf2-3943f5e03922-kube-api-access-b2zkv\") pod \"222e7c04-54d7-4429-9bf2-3943f5e03922\" (UID: \"222e7c04-54d7-4429-9bf2-3943f5e03922\") "
Nov 02 13:59:58.971101 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:58.971072    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-openshift-ca" pod "222e7c04-54d7-4429-9bf2-3943f5e03922" (UID: "222e7c04-54d7-4429-9bf2-3943f5e03922"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 13:59:58.973402 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:58.973376    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222e7c04-54d7-4429-9bf2-3943f5e03922-kube-api-access-b2zkv" pod "222e7c04-54d7-4429-9bf2-3943f5e03922" (UID: "222e7c04-54d7-4429-9bf2-3943f5e03922"). InnerVolumeSpecName "kube-api-access-b2zkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 13:59:58.975244 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:58.975222    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-xks-ca" pod "222e7c04-54d7-4429-9bf2-3943f5e03922" (UID: "222e7c04-54d7-4429-9bf2-3943f5e03922"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 13:59:59.071981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:59.071897    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2zkv\" (UniqueName: \"kubernetes.io/projected/222e7c04-54d7-4429-9bf2-3943f5e03922-kube-api-access-b2zkv\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:59:59.071981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:59.071934    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:59:59.071981 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:59.071954    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/222e7c04-54d7-4429-9bf2-3943f5e03922-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 13:59:59.479261 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:59.479223    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs"
Nov 02 13:59:59.479261 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:59.479249    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6005186d-mjkxs" event={"ID":"222e7c04-54d7-4429-9bf2-3943f5e03922","Type":"ContainerDied","Data":"b616b29496f6e994c519baf647bf798063572d0ea98324697c18fef16db02929"}
Nov 02 13:59:59.479508 ip-10-0-134-229 kubenswrapper[5879]: I1102 13:59:59.479301    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b616b29496f6e994c519baf647bf798063572d0ea98324697c18fef16db02929"
Nov 02 14:00:00.139624 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.139587    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"]
Nov 02 14:00:00.140279 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.140257    5879 cpu_manager.go:414] "containerMap: removing container" podUID="222e7c04-54d7-4429-9bf2-3943f5e03922" containerName="revoke-keys"
Nov 02 14:00:00.140386 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.140282    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222e7c04-54d7-4429-9bf2-3943f5e03922" containerName="revoke-keys"
Nov 02 14:00:00.140463 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.140415    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="222e7c04-54d7-4429-9bf2-3943f5e03922" containerName="revoke-keys"
Nov 02 14:00:00.143947 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.143921    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"
Nov 02 14:00:00.163934 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.163904    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"]
Nov 02 14:00:00.175430 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.175379    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 14:00:00.182472 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.182441    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trdjx\" (UniqueName: \"kubernetes.io/projected/17382d18-a3a1-46f4-9ffd-149bce13b413-kube-api-access-trdjx\") pod \"maas-api-key-cleanup-29893800-d6hcv\" (UID: \"17382d18-a3a1-46f4-9ffd-149bce13b413\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"
Nov 02 14:00:00.182612 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.182584    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17382d18-a3a1-46f4-9ffd-149bce13b413-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893800-d6hcv\" (UID: \"17382d18-a3a1-46f4-9ffd-149bce13b413\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"
Nov 02 14:00:00.284216 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.284170    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trdjx\" (UniqueName: \"kubernetes.io/projected/17382d18-a3a1-46f4-9ffd-149bce13b413-kube-api-access-trdjx\") pod \"maas-api-key-cleanup-29893800-d6hcv\" (UID: \"17382d18-a3a1-46f4-9ffd-149bce13b413\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"
Nov 02 14:00:00.284439 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.284239    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17382d18-a3a1-46f4-9ffd-149bce13b413-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893800-d6hcv\" (UID: \"17382d18-a3a1-46f4-9ffd-149bce13b413\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"
Nov 02 14:00:00.284971 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.284943    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17382d18-a3a1-46f4-9ffd-149bce13b413-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893800-d6hcv\" (UID: \"17382d18-a3a1-46f4-9ffd-149bce13b413\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"
Nov 02 14:00:00.310320 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.310283    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trdjx\" (UniqueName: \"kubernetes.io/projected/17382d18-a3a1-46f4-9ffd-149bce13b413-kube-api-access-trdjx\") pod \"maas-api-key-cleanup-29893800-d6hcv\" (UID: \"17382d18-a3a1-46f4-9ffd-149bce13b413\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"
Nov 02 14:00:00.457564 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.457479    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"
Nov 02 14:00:00.759831 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:00.759799    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"]
Nov 02 14:00:00.759978 ip-10-0-134-229 kubenswrapper[5879]: W1102 14:00:00.759930    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17382d18_a3a1_46f4_9ffd_149bce13b413.slice/crio-0dcdfcb3b8136f132457068a569598422d57ac0f3ffabdc99edb13901f14ba71 WatchSource:0}: Error finding container 0dcdfcb3b8136f132457068a569598422d57ac0f3ffabdc99edb13901f14ba71: Status 404 returned error can't find the container with id 0dcdfcb3b8136f132457068a569598422d57ac0f3ffabdc99edb13901f14ba71
Nov 02 14:00:01.491784 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:01.491746    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv" event={"ID":"17382d18-a3a1-46f4-9ffd-149bce13b413","Type":"ContainerStarted","Data":"94e4b2d71c15c3629beed98c5e4125255c8853634b1c1c7fd259601d8355b2f3"}
Nov 02 14:00:01.492293 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:01.491805    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv" event={"ID":"17382d18-a3a1-46f4-9ffd-149bce13b413","Type":"ContainerStarted","Data":"0dcdfcb3b8136f132457068a569598422d57ac0f3ffabdc99edb13901f14ba71"}
Nov 02 14:00:01.537110 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:01.537052    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv" podStartSLOduration=1.537036039 podStartE2EDuration="1.537036039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:01.53534951 +0000 UTC m=+5547.642052360" watchObservedRunningTime="2026-11-02 14:00:01.537036039 +0000 UTC m=+5547.643738890"
Nov 02 14:00:02.500413 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:02.500371    5879 generic.go:356] "Generic (PLEG): container finished" podID="17382d18-a3a1-46f4-9ffd-149bce13b413" containerID="94e4b2d71c15c3629beed98c5e4125255c8853634b1c1c7fd259601d8355b2f3" exitCode=60
Nov 02 14:00:02.500928 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:02.500463    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv" event={"ID":"17382d18-a3a1-46f4-9ffd-149bce13b413","Type":"ContainerDied","Data":"94e4b2d71c15c3629beed98c5e4125255c8853634b1c1c7fd259601d8355b2f3"}
Nov 02 14:00:02.500928 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:02.500728    5879 scope.go:122] "RemoveContainer" containerID="94e4b2d71c15c3629beed98c5e4125255c8853634b1c1c7fd259601d8355b2f3"
Nov 02 14:00:03.508643 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:03.508606    5879 generic.go:356] "Generic (PLEG): container finished" podID="17382d18-a3a1-46f4-9ffd-149bce13b413" containerID="b50ac1f8666b63ebd7853ec159a996456a73bc1c05bc552fd2f74f8c75271152" exitCode=60
Nov 02 14:00:03.509180 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:03.508670    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv" event={"ID":"17382d18-a3a1-46f4-9ffd-149bce13b413","Type":"ContainerDied","Data":"b50ac1f8666b63ebd7853ec159a996456a73bc1c05bc552fd2f74f8c75271152"}
Nov 02 14:00:03.509180 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:03.508718    5879 scope.go:122] "RemoveContainer" containerID="94e4b2d71c15c3629beed98c5e4125255c8853634b1c1c7fd259601d8355b2f3"
Nov 02 14:00:03.509180 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:03.508973    5879 scope.go:122] "RemoveContainer" containerID="b50ac1f8666b63ebd7853ec159a996456a73bc1c05bc552fd2f74f8c75271152"
Nov 02 14:00:03.509320 ip-10-0-134-229 kubenswrapper[5879]: E1102 14:00:03.509289    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893800-d6hcv_odh-ai-gateway-infra(17382d18-a3a1-46f4-9ffd-149bce13b413)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv" podUID="17382d18-a3a1-46f4-9ffd-149bce13b413"
Nov 02 14:00:04.515634 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:04.515604    5879 scope.go:122] "RemoveContainer" containerID="b50ac1f8666b63ebd7853ec159a996456a73bc1c05bc552fd2f74f8c75271152"
Nov 02 14:00:04.516090 ip-10-0-134-229 kubenswrapper[5879]: E1102 14:00:04.515853    5879 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893800-d6hcv_odh-ai-gateway-infra(17382d18-a3a1-46f4-9ffd-149bce13b413)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv" podUID="17382d18-a3a1-46f4-9ffd-149bce13b413"
Nov 02 14:00:14.821736 ip-10-0-134-229 kubenswrapper[5879]: E1102 14:00:14.821697    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 14:00:17.009720 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:17.009681    5879 scope.go:122] "RemoveContainer" containerID="b50ac1f8666b63ebd7853ec159a996456a73bc1c05bc552fd2f74f8c75271152"
Nov 02 14:00:17.598474 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:17.598373    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv" event={"ID":"17382d18-a3a1-46f4-9ffd-149bce13b413","Type":"ContainerStarted","Data":"14824f7fbdb6bc2238e4e9ef4ec5569be151e1a624bc9554027ea17806349363"}
Nov 02 14:00:18.654784 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:18.654745    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"]
Nov 02 14:00:18.659359 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:18.659307    5879 generic.go:356] "Generic (PLEG): container finished" podID="17382d18-a3a1-46f4-9ffd-149bce13b413" containerID="14824f7fbdb6bc2238e4e9ef4ec5569be151e1a624bc9554027ea17806349363" exitCode=60
Nov 02 14:00:18.659836 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:18.659791    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv" event={"ID":"17382d18-a3a1-46f4-9ffd-149bce13b413","Type":"ContainerDied","Data":"14824f7fbdb6bc2238e4e9ef4ec5569be151e1a624bc9554027ea17806349363"}
Nov 02 14:00:18.660129 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:18.660092    5879 scope.go:122] "RemoveContainer" containerID="b50ac1f8666b63ebd7853ec159a996456a73bc1c05bc552fd2f74f8c75271152"
Nov 02 14:00:18.989637 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:18.989292    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"
Nov 02 14:00:19.091548 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:19.090358    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17382d18-a3a1-46f4-9ffd-149bce13b413-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17382d18-a3a1-46f4-9ffd-149bce13b413-maas-api-service-ca\") pod \"17382d18-a3a1-46f4-9ffd-149bce13b413\" (UID: \"17382d18-a3a1-46f4-9ffd-149bce13b413\") "
Nov 02 14:00:19.091548 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:19.090608    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17382d18-a3a1-46f4-9ffd-149bce13b413-kube-api-access-trdjx\" (UniqueName: \"kubernetes.io/projected/17382d18-a3a1-46f4-9ffd-149bce13b413-kube-api-access-trdjx\") pod \"17382d18-a3a1-46f4-9ffd-149bce13b413\" (UID: \"17382d18-a3a1-46f4-9ffd-149bce13b413\") "
Nov 02 14:00:19.092486 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:19.092448    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17382d18-a3a1-46f4-9ffd-149bce13b413-maas-api-service-ca" pod "17382d18-a3a1-46f4-9ffd-149bce13b413" (UID: "17382d18-a3a1-46f4-9ffd-149bce13b413"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:19.154058 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:19.130712    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17382d18-a3a1-46f4-9ffd-149bce13b413-kube-api-access-trdjx" pod "17382d18-a3a1-46f4-9ffd-149bce13b413" (UID: "17382d18-a3a1-46f4-9ffd-149bce13b413"). InnerVolumeSpecName "kube-api-access-trdjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:19.195107 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:19.194763    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trdjx\" (UniqueName: \"kubernetes.io/projected/17382d18-a3a1-46f4-9ffd-149bce13b413-kube-api-access-trdjx\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:00:19.195107 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:19.194812    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17382d18-a3a1-46f4-9ffd-149bce13b413-maas-api-service-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:00:19.666972 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:19.666936    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"
Nov 02 14:00:19.667484 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:19.666941    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv" event={"ID":"17382d18-a3a1-46f4-9ffd-149bce13b413","Type":"ContainerDied","Data":"0dcdfcb3b8136f132457068a569598422d57ac0f3ffabdc99edb13901f14ba71"}
Nov 02 14:00:19.667484 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:19.667059    5879 scope.go:122] "RemoveContainer" containerID="14824f7fbdb6bc2238e4e9ef4ec5569be151e1a624bc9554027ea17806349363"
Nov 02 14:00:19.746020 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:19.745540    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"]
Nov 02 14:00:19.756048 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:19.755590    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893800-d6hcv"]
Nov 02 14:00:20.015107 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:20.015065    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17382d18-a3a1-46f4-9ffd-149bce13b413" path="/var/lib/kubelet/pods/17382d18-a3a1-46f4-9ffd-149bce13b413/volumes"
Nov 02 14:00:26.231730 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.231689    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"]
Nov 02 14:00:26.232297 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.232160    5879 cpu_manager.go:414] "containerMap: removing container" podUID="17382d18-a3a1-46f4-9ffd-149bce13b413" containerName="cleanup"
Nov 02 14:00:26.232297 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.232180    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17382d18-a3a1-46f4-9ffd-149bce13b413" containerName="cleanup"
Nov 02 14:00:26.232297 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.232249    5879 cpu_manager.go:414] "containerMap: removing container" podUID="17382d18-a3a1-46f4-9ffd-149bce13b413" containerName="cleanup"
Nov 02 14:00:26.232297 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.232259    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17382d18-a3a1-46f4-9ffd-149bce13b413" containerName="cleanup"
Nov 02 14:00:26.232561 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.232370    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="17382d18-a3a1-46f4-9ffd-149bce13b413" containerName="cleanup"
Nov 02 14:00:26.232561 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.232385    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="17382d18-a3a1-46f4-9ffd-149bce13b413" containerName="cleanup"
Nov 02 14:00:26.236748 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.236725    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:26.256221 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.256188    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"]
Nov 02 14:00:26.267277 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.267244    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:26.267498 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.267302    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 14:00:26.376900 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.376859    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxbwr\" (UniqueName: \"kubernetes.io/projected/e28783d4-5446-4802-a9e1-134b867cd015-kube-api-access-hxbwr\") pod \"maas-api-revoke-sub-4fe77a7a-p2lsr\" (UID: \"e28783d4-5446-4802-a9e1-134b867cd015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:26.377146 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.377039    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fe77a7a-p2lsr\" (UID: \"e28783d4-5446-4802-a9e1-134b867cd015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:26.377146 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.377078    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fe77a7a-p2lsr\" (UID: \"e28783d4-5446-4802-a9e1-134b867cd015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:26.478344 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.478297    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fe77a7a-p2lsr\" (UID: \"e28783d4-5446-4802-a9e1-134b867cd015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:26.478569 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.478372    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fe77a7a-p2lsr\" (UID: \"e28783d4-5446-4802-a9e1-134b867cd015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:26.478569 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.478420    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbwr\" (UniqueName: \"kubernetes.io/projected/e28783d4-5446-4802-a9e1-134b867cd015-kube-api-access-hxbwr\") pod \"maas-api-revoke-sub-4fe77a7a-p2lsr\" (UID: \"e28783d4-5446-4802-a9e1-134b867cd015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:26.479151 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.479067    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fe77a7a-p2lsr\" (UID: \"e28783d4-5446-4802-a9e1-134b867cd015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:26.480922 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.480900    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fe77a7a-p2lsr\" (UID: \"e28783d4-5446-4802-a9e1-134b867cd015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:26.504777 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.504694    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxbwr\" (UniqueName: \"kubernetes.io/projected/e28783d4-5446-4802-a9e1-134b867cd015-kube-api-access-hxbwr\") pod \"maas-api-revoke-sub-4fe77a7a-p2lsr\" (UID: \"e28783d4-5446-4802-a9e1-134b867cd015\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:26.551030 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.550959    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:26.856645 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:26.856596    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"]
Nov 02 14:00:26.856955 ip-10-0-134-229 kubenswrapper[5879]: W1102 14:00:26.856929    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28783d4_5446_4802_a9e1_134b867cd015.slice/crio-2356d19ee995c598fbab87e0e45ccef7b8c77bb2a7536376b5eac03a97385c2c WatchSource:0}: Error finding container 2356d19ee995c598fbab87e0e45ccef7b8c77bb2a7536376b5eac03a97385c2c: Status 404 returned error can't find the container with id 2356d19ee995c598fbab87e0e45ccef7b8c77bb2a7536376b5eac03a97385c2c
Nov 02 14:00:27.714987 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:27.714939    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr" event={"ID":"e28783d4-5446-4802-a9e1-134b867cd015","Type":"ContainerStarted","Data":"b68be5928c49c8a2bbeb13369672cd4e9544caacab6ab6cd95701c741f66c85e"}
Nov 02 14:00:27.715446 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:27.715035    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr" event={"ID":"e28783d4-5446-4802-a9e1-134b867cd015","Type":"ContainerStarted","Data":"2356d19ee995c598fbab87e0e45ccef7b8c77bb2a7536376b5eac03a97385c2c"}
Nov 02 14:00:28.752019 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:28.751916    5879 generic.go:356] "Generic (PLEG): container finished" podID="e28783d4-5446-4802-a9e1-134b867cd015" containerID="b68be5928c49c8a2bbeb13369672cd4e9544caacab6ab6cd95701c741f66c85e" exitCode=0
Nov 02 14:00:28.752613 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:28.752187    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr" event={"ID":"e28783d4-5446-4802-a9e1-134b867cd015","Type":"ContainerDied","Data":"b68be5928c49c8a2bbeb13369672cd4e9544caacab6ab6cd95701c741f66c85e"}
Nov 02 14:00:30.073617 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.073591    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:30.161048 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.160504    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-xks-ca\") pod \"e28783d4-5446-4802-a9e1-134b867cd015\" (UID: \"e28783d4-5446-4802-a9e1-134b867cd015\") "
Nov 02 14:00:30.161048 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.160620    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28783d4-5446-4802-a9e1-134b867cd015-kube-api-access-hxbwr\" (UniqueName: \"kubernetes.io/projected/e28783d4-5446-4802-a9e1-134b867cd015-kube-api-access-hxbwr\") pod \"e28783d4-5446-4802-a9e1-134b867cd015\" (UID: \"e28783d4-5446-4802-a9e1-134b867cd015\") "
Nov 02 14:00:30.161048 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.160672    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-openshift-ca\") pod \"e28783d4-5446-4802-a9e1-134b867cd015\" (UID: \"e28783d4-5446-4802-a9e1-134b867cd015\") "
Nov 02 14:00:30.162801 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.162765    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-openshift-ca" pod "e28783d4-5446-4802-a9e1-134b867cd015" (UID: "e28783d4-5446-4802-a9e1-134b867cd015"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:30.164281 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.164254    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28783d4-5446-4802-a9e1-134b867cd015-kube-api-access-hxbwr" pod "e28783d4-5446-4802-a9e1-134b867cd015" (UID: "e28783d4-5446-4802-a9e1-134b867cd015"). InnerVolumeSpecName "kube-api-access-hxbwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:30.165242 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.165218    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-xks-ca" pod "e28783d4-5446-4802-a9e1-134b867cd015" (UID: "e28783d4-5446-4802-a9e1-134b867cd015"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:30.261575 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.261536    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:00:30.261575 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.261572    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e28783d4-5446-4802-a9e1-134b867cd015-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:00:30.261575 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.261582    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxbwr\" (UniqueName: \"kubernetes.io/projected/e28783d4-5446-4802-a9e1-134b867cd015-kube-api-access-hxbwr\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:00:30.768524 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.768491    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr"
Nov 02 14:00:30.768735 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.768531    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe77a7a-p2lsr" event={"ID":"e28783d4-5446-4802-a9e1-134b867cd015","Type":"ContainerDied","Data":"2356d19ee995c598fbab87e0e45ccef7b8c77bb2a7536376b5eac03a97385c2c"}
Nov 02 14:00:30.768735 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:30.768582    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2356d19ee995c598fbab87e0e45ccef7b8c77bb2a7536376b5eac03a97385c2c"
Nov 02 14:00:36.575754 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.575723    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"]
Nov 02 14:00:36.576238 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.576222    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e28783d4-5446-4802-a9e1-134b867cd015" containerName="revoke-keys"
Nov 02 14:00:36.576289 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.576242    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28783d4-5446-4802-a9e1-134b867cd015" containerName="revoke-keys"
Nov 02 14:00:36.576289 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.576265    5879 cpu_manager.go:414] "containerMap: removing container" podUID="17382d18-a3a1-46f4-9ffd-149bce13b413" containerName="cleanup"
Nov 02 14:00:36.576289 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.576271    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17382d18-a3a1-46f4-9ffd-149bce13b413" containerName="cleanup"
Nov 02 14:00:36.576398 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.576362    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28783d4-5446-4802-a9e1-134b867cd015" containerName="revoke-keys"
Nov 02 14:00:36.576398 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.576378    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="17382d18-a3a1-46f4-9ffd-149bce13b413" containerName="cleanup"
Nov 02 14:00:36.579789 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.579772    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:36.610798 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.610751    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:36.611065 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.610932    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 14:00:36.614500 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.614467    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"]
Nov 02 14:00:36.629957 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.629921    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda01314-bsvwv\" (UID: \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:36.630173 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.630033    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mj5n\" (UniqueName: \"kubernetes.io/projected/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-kube-api-access-8mj5n\") pod \"maas-api-revoke-sub-eda01314-bsvwv\" (UID: \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:36.630228 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.630195    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda01314-bsvwv\" (UID: \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:36.731370 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.731307    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda01314-bsvwv\" (UID: \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:36.731370 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.731370    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mj5n\" (UniqueName: \"kubernetes.io/projected/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-kube-api-access-8mj5n\") pod \"maas-api-revoke-sub-eda01314-bsvwv\" (UID: \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:36.731673 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.731523    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda01314-bsvwv\" (UID: \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:36.732230 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.732200    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eda01314-bsvwv\" (UID: \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:36.734166 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.734144    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eda01314-bsvwv\" (UID: \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:36.757524 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.757498    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mj5n\" (UniqueName: \"kubernetes.io/projected/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-kube-api-access-8mj5n\") pod \"maas-api-revoke-sub-eda01314-bsvwv\" (UID: \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:36.898406 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:36.898365    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:37.199602 ip-10-0-134-229 kubenswrapper[5879]: W1102 14:00:37.199571    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0baebe0_7263_4aec_9fbd_96b0a5d5fa2c.slice/crio-e7361f089cb1a575e2d34ad02ad15f94c4d54bf5150564d6ca0645feace10fe4 WatchSource:0}: Error finding container e7361f089cb1a575e2d34ad02ad15f94c4d54bf5150564d6ca0645feace10fe4: Status 404 returned error can't find the container with id e7361f089cb1a575e2d34ad02ad15f94c4d54bf5150564d6ca0645feace10fe4
Nov 02 14:00:37.199788 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:37.199761    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"]
Nov 02 14:00:37.809144 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:37.809038    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv" event={"ID":"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c","Type":"ContainerStarted","Data":"bcba56980fe950f46c96d615aedf0ae102c2da6704c0428731474eff5718f8e0"}
Nov 02 14:00:37.809144 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:37.809107    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv" event={"ID":"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c","Type":"ContainerStarted","Data":"e7361f089cb1a575e2d34ad02ad15f94c4d54bf5150564d6ca0645feace10fe4"}
Nov 02 14:00:37.855004 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:37.854947    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv" podStartSLOduration=1.85493215 podStartE2EDuration="1.85493215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:00:37.854159608 +0000 UTC m=+5583.960862459" watchObservedRunningTime="2026-11-02 14:00:37.85493215 +0000 UTC m=+5583.961635038"
Nov 02 14:00:38.818969 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:38.818931    5879 generic.go:356] "Generic (PLEG): container finished" podID="e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c" containerID="bcba56980fe950f46c96d615aedf0ae102c2da6704c0428731474eff5718f8e0" exitCode=0
Nov 02 14:00:38.819422 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:38.819031    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv" event={"ID":"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c","Type":"ContainerDied","Data":"bcba56980fe950f46c96d615aedf0ae102c2da6704c0428731474eff5718f8e0"}
Nov 02 14:00:40.201023 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.196670    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:40.273837 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.273792    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-openshift-ca\") pod \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\" (UID: \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\") "
Nov 02 14:00:40.273837 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.273834    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-xks-ca\") pod \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\" (UID: \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\") "
Nov 02 14:00:40.274174 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.273944    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-kube-api-access-8mj5n\" (UniqueName: \"kubernetes.io/projected/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-kube-api-access-8mj5n\") pod \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\" (UID: \"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c\") "
Nov 02 14:00:40.274263 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.274237    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-openshift-ca" pod "e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c" (UID: "e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:40.274338 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.274307    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:00:40.276215 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.276189    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-kube-api-access-8mj5n" pod "e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c" (UID: "e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c"). InnerVolumeSpecName "kube-api-access-8mj5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:40.276215 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.276193    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-xks-ca" pod "e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c" (UID: "e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:40.374983 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.374919    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:00:40.374983 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.374974    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mj5n\" (UniqueName: \"kubernetes.io/projected/e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c-kube-api-access-8mj5n\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:00:40.852111 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.852075    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv"
Nov 02 14:00:40.852324 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.852113    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eda01314-bsvwv" event={"ID":"e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c","Type":"ContainerDied","Data":"e7361f089cb1a575e2d34ad02ad15f94c4d54bf5150564d6ca0645feace10fe4"}
Nov 02 14:00:40.852324 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:40.852168    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7361f089cb1a575e2d34ad02ad15f94c4d54bf5150564d6ca0645feace10fe4"
Nov 02 14:00:44.528752 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:44.528661    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:44.820441 ip-10-0-134-229 kubenswrapper[5879]: E1102 14:00:44.820343    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 14:00:45.371279 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:45.371238    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 14:00:46.933146 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:46.933111    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"]
Nov 02 14:00:46.933724 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:46.933574    5879 cpu_manager.go:414] "containerMap: removing container" podUID="e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c" containerName="revoke-keys"
Nov 02 14:00:46.933724 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:46.933588    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c" containerName="revoke-keys"
Nov 02 14:00:46.933724 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:46.933696    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c" containerName="revoke-keys"
Nov 02 14:00:46.937187 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:46.937167    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:46.953910 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:46.953875    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:46.956366 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:46.956336    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"]
Nov 02 14:00:46.968044 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:46.968013    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 14:00:47.050829 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:47.050790    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf436c7-cc899\" (UID: \"ecd90196-10cd-4640-bd9c-318dac558ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:47.051090 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:47.050902    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf436c7-cc899\" (UID: \"ecd90196-10cd-4640-bd9c-318dac558ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:47.051090 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:47.050956    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgbz\" (UniqueName: \"kubernetes.io/projected/ecd90196-10cd-4640-bd9c-318dac558ef5-kube-api-access-7vgbz\") pod \"maas-api-revoke-sub-daf436c7-cc899\" (UID: \"ecd90196-10cd-4640-bd9c-318dac558ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:47.152161 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:47.152113    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgbz\" (UniqueName: \"kubernetes.io/projected/ecd90196-10cd-4640-bd9c-318dac558ef5-kube-api-access-7vgbz\") pod \"maas-api-revoke-sub-daf436c7-cc899\" (UID: \"ecd90196-10cd-4640-bd9c-318dac558ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:47.152350 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:47.152276    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf436c7-cc899\" (UID: \"ecd90196-10cd-4640-bd9c-318dac558ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:47.152350 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:47.152332    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf436c7-cc899\" (UID: \"ecd90196-10cd-4640-bd9c-318dac558ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:47.153058 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:47.153020    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daf436c7-cc899\" (UID: \"ecd90196-10cd-4640-bd9c-318dac558ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:47.154867 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:47.154837    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daf436c7-cc899\" (UID: \"ecd90196-10cd-4640-bd9c-318dac558ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:47.184864 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:47.184784    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgbz\" (UniqueName: \"kubernetes.io/projected/ecd90196-10cd-4640-bd9c-318dac558ef5-kube-api-access-7vgbz\") pod \"maas-api-revoke-sub-daf436c7-cc899\" (UID: \"ecd90196-10cd-4640-bd9c-318dac558ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:47.250886 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:47.250844    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:47.558177 ip-10-0-134-229 kubenswrapper[5879]: W1102 14:00:47.558135    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd90196_10cd_4640_bd9c_318dac558ef5.slice/crio-9c44716863fd778ed0f3306213cc672a9cafb22e3be46605f12eabb26486f483 WatchSource:0}: Error finding container 9c44716863fd778ed0f3306213cc672a9cafb22e3be46605f12eabb26486f483: Status 404 returned error can't find the container with id 9c44716863fd778ed0f3306213cc672a9cafb22e3be46605f12eabb26486f483
Nov 02 14:00:47.558792 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:47.558760    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"]
Nov 02 14:00:47.888051 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:47.888019    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899" event={"ID":"ecd90196-10cd-4640-bd9c-318dac558ef5","Type":"ContainerStarted","Data":"9c44716863fd778ed0f3306213cc672a9cafb22e3be46605f12eabb26486f483"}
Nov 02 14:00:48.896486 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:48.896452    5879 generic.go:356] "Generic (PLEG): container finished" podID="ecd90196-10cd-4640-bd9c-318dac558ef5" containerID="a5a3675daad9ad17ef2dd0ef6ec6e540e84c93e67d874a7df3bab066e4734fa3" exitCode=0
Nov 02 14:00:48.896958 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:48.896545    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899" event={"ID":"ecd90196-10cd-4640-bd9c-318dac558ef5","Type":"ContainerDied","Data":"a5a3675daad9ad17ef2dd0ef6ec6e540e84c93e67d874a7df3bab066e4734fa3"}
Nov 02 14:00:50.275155 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.275129    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:50.441432 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.441393    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-xks-ca\") pod \"ecd90196-10cd-4640-bd9c-318dac558ef5\" (UID: \"ecd90196-10cd-4640-bd9c-318dac558ef5\") "
Nov 02 14:00:50.441774 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.441750    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-openshift-ca\") pod \"ecd90196-10cd-4640-bd9c-318dac558ef5\" (UID: \"ecd90196-10cd-4640-bd9c-318dac558ef5\") "
Nov 02 14:00:50.442038 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.442018    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd90196-10cd-4640-bd9c-318dac558ef5-kube-api-access-7vgbz\" (UniqueName: \"kubernetes.io/projected/ecd90196-10cd-4640-bd9c-318dac558ef5-kube-api-access-7vgbz\") pod \"ecd90196-10cd-4640-bd9c-318dac558ef5\" (UID: \"ecd90196-10cd-4640-bd9c-318dac558ef5\") "
Nov 02 14:00:50.452045 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.450720    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd90196-10cd-4640-bd9c-318dac558ef5-kube-api-access-7vgbz" pod "ecd90196-10cd-4640-bd9c-318dac558ef5" (UID: "ecd90196-10cd-4640-bd9c-318dac558ef5"). InnerVolumeSpecName "kube-api-access-7vgbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:00:50.452045 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.451132    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-openshift-ca" pod "ecd90196-10cd-4640-bd9c-318dac558ef5" (UID: "ecd90196-10cd-4640-bd9c-318dac558ef5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:00:50.452799 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.452772    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-xks-ca" pod "ecd90196-10cd-4640-bd9c-318dac558ef5" (UID: "ecd90196-10cd-4640-bd9c-318dac558ef5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:00:50.544917 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.544759    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:00:50.544917 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.544808    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vgbz\" (UniqueName: \"kubernetes.io/projected/ecd90196-10cd-4640-bd9c-318dac558ef5-kube-api-access-7vgbz\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:00:50.544917 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.544826    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecd90196-10cd-4640-bd9c-318dac558ef5-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:00:50.930532 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.930492    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899" event={"ID":"ecd90196-10cd-4640-bd9c-318dac558ef5","Type":"ContainerDied","Data":"9c44716863fd778ed0f3306213cc672a9cafb22e3be46605f12eabb26486f483"}
Nov 02 14:00:50.930532 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.930527    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daf436c7-cc899"
Nov 02 14:00:50.930768 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:50.930545    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c44716863fd778ed0f3306213cc672a9cafb22e3be46605f12eabb26486f483"
Nov 02 14:00:57.291727 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.291693    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"]
Nov 02 14:00:57.292229 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.292180    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ecd90196-10cd-4640-bd9c-318dac558ef5" containerName="revoke-keys"
Nov 02 14:00:57.292229 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.292192    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd90196-10cd-4640-bd9c-318dac558ef5" containerName="revoke-keys"
Nov 02 14:00:57.292328 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.292256    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd90196-10cd-4640-bd9c-318dac558ef5" containerName="revoke-keys"
Nov 02 14:00:57.295872 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.295845    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:00:57.314451 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.314411    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6bc25c9-jspdr\" (UID: \"725153ba-0bf2-4c41-bc02-6f5db080c686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:00:57.314788 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.314768    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6bc25c9-jspdr\" (UID: \"725153ba-0bf2-4c41-bc02-6f5db080c686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:00:57.315055 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.315035    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkmkc\" (UniqueName: \"kubernetes.io/projected/725153ba-0bf2-4c41-bc02-6f5db080c686-kube-api-access-lkmkc\") pod \"maas-api-revoke-sub-f6bc25c9-jspdr\" (UID: \"725153ba-0bf2-4c41-bc02-6f5db080c686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:00:57.315634 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.315609    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"]
Nov 02 14:00:57.326838 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.326803    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:00:57.327026 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.326891    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 14:00:57.416050 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.415983    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6bc25c9-jspdr\" (UID: \"725153ba-0bf2-4c41-bc02-6f5db080c686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:00:57.416286 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.416104    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkmkc\" (UniqueName: \"kubernetes.io/projected/725153ba-0bf2-4c41-bc02-6f5db080c686-kube-api-access-lkmkc\") pod \"maas-api-revoke-sub-f6bc25c9-jspdr\" (UID: \"725153ba-0bf2-4c41-bc02-6f5db080c686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:00:57.416286 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.416183    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6bc25c9-jspdr\" (UID: \"725153ba-0bf2-4c41-bc02-6f5db080c686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:00:57.416829 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.416796    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6bc25c9-jspdr\" (UID: \"725153ba-0bf2-4c41-bc02-6f5db080c686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:00:57.418742 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.418714    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6bc25c9-jspdr\" (UID: \"725153ba-0bf2-4c41-bc02-6f5db080c686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:00:57.441999 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.441969    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkmkc\" (UniqueName: \"kubernetes.io/projected/725153ba-0bf2-4c41-bc02-6f5db080c686-kube-api-access-lkmkc\") pod \"maas-api-revoke-sub-f6bc25c9-jspdr\" (UID: \"725153ba-0bf2-4c41-bc02-6f5db080c686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:00:57.610110 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.610022    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:00:57.911594 ip-10-0-134-229 kubenswrapper[5879]: W1102 14:00:57.911548    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725153ba_0bf2_4c41_bc02_6f5db080c686.slice/crio-3871909405e1de0b0feb77b18ecdb5ee75d57331a8d3e7726c12d7484a7e6ac1 WatchSource:0}: Error finding container 3871909405e1de0b0feb77b18ecdb5ee75d57331a8d3e7726c12d7484a7e6ac1: Status 404 returned error can't find the container with id 3871909405e1de0b0feb77b18ecdb5ee75d57331a8d3e7726c12d7484a7e6ac1
Nov 02 14:00:57.912035 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.911933    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"]
Nov 02 14:00:57.967718 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:57.967680    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr" event={"ID":"725153ba-0bf2-4c41-bc02-6f5db080c686","Type":"ContainerStarted","Data":"3871909405e1de0b0feb77b18ecdb5ee75d57331a8d3e7726c12d7484a7e6ac1"}
Nov 02 14:00:58.978617 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:58.978510    5879 generic.go:356] "Generic (PLEG): container finished" podID="725153ba-0bf2-4c41-bc02-6f5db080c686" containerID="8dd12fff253d88f9000cc7e9498780a7fa689fc77390066812f5c5c3ac360a85" exitCode=0
Nov 02 14:00:58.979160 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:00:58.978607    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr" event={"ID":"725153ba-0bf2-4c41-bc02-6f5db080c686","Type":"ContainerDied","Data":"8dd12fff253d88f9000cc7e9498780a7fa689fc77390066812f5c5c3ac360a85"}
Nov 02 14:01:00.380552 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:00.380454    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:01:00.464397 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:00.464358    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-xks-ca\") pod \"725153ba-0bf2-4c41-bc02-6f5db080c686\" (UID: \"725153ba-0bf2-4c41-bc02-6f5db080c686\") "
Nov 02 14:01:00.464706 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:00.464678    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/725153ba-0bf2-4c41-bc02-6f5db080c686-kube-api-access-lkmkc\" (UniqueName: \"kubernetes.io/projected/725153ba-0bf2-4c41-bc02-6f5db080c686-kube-api-access-lkmkc\") pod \"725153ba-0bf2-4c41-bc02-6f5db080c686\" (UID: \"725153ba-0bf2-4c41-bc02-6f5db080c686\") "
Nov 02 14:01:00.464834 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:00.464715    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-openshift-ca\") pod \"725153ba-0bf2-4c41-bc02-6f5db080c686\" (UID: \"725153ba-0bf2-4c41-bc02-6f5db080c686\") "
Nov 02 14:01:00.465406 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:00.465379    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-openshift-ca" pod "725153ba-0bf2-4c41-bc02-6f5db080c686" (UID: "725153ba-0bf2-4c41-bc02-6f5db080c686"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:00.468747 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:00.468716    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-xks-ca" pod "725153ba-0bf2-4c41-bc02-6f5db080c686" (UID: "725153ba-0bf2-4c41-bc02-6f5db080c686"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:01:00.469173 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:00.469149    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725153ba-0bf2-4c41-bc02-6f5db080c686-kube-api-access-lkmkc" pod "725153ba-0bf2-4c41-bc02-6f5db080c686" (UID: "725153ba-0bf2-4c41-bc02-6f5db080c686"). InnerVolumeSpecName "kube-api-access-lkmkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:00.565768 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:00.565715    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:01:00.565768 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:00.565759    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkmkc\" (UniqueName: \"kubernetes.io/projected/725153ba-0bf2-4c41-bc02-6f5db080c686-kube-api-access-lkmkc\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:01:00.565768 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:00.565774    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/725153ba-0bf2-4c41-bc02-6f5db080c686-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:01:00.856757 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:00.856669    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-m2dn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:01:00.857218 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:00.857167    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-m2dn7" podUID="e1ec3a45-ae38-4297-a894-27438a341dcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:01.040477 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:01.040438    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr" event={"ID":"725153ba-0bf2-4c41-bc02-6f5db080c686","Type":"ContainerDied","Data":"3871909405e1de0b0feb77b18ecdb5ee75d57331a8d3e7726c12d7484a7e6ac1"}
Nov 02 14:01:01.040759 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:01.040740    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3871909405e1de0b0feb77b18ecdb5ee75d57331a8d3e7726c12d7484a7e6ac1"
Nov 02 14:01:01.040939 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:01.040925    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6bc25c9-jspdr"
Nov 02 14:01:07.709817 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.709783    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"]
Nov 02 14:01:07.710287 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.710273    5879 cpu_manager.go:414] "containerMap: removing container" podUID="725153ba-0bf2-4c41-bc02-6f5db080c686" containerName="revoke-keys"
Nov 02 14:01:07.710343 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.710288    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="725153ba-0bf2-4c41-bc02-6f5db080c686" containerName="revoke-keys"
Nov 02 14:01:07.710381 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.710368    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="725153ba-0bf2-4c41-bc02-6f5db080c686" containerName="revoke-keys"
Nov 02 14:01:07.713936 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.713916    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:07.726059 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.726032    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dafa3531-psrvh\" (UID: \"0f84c2a9-a227-45bc-8690-b610658bff5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:07.726231 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.726195    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dafa3531-psrvh\" (UID: \"0f84c2a9-a227-45bc-8690-b610658bff5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:07.726299 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.726271    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj5pk\" (UniqueName: \"kubernetes.io/projected/0f84c2a9-a227-45bc-8690-b610658bff5f-kube-api-access-cj5pk\") pod \"maas-api-revoke-sub-dafa3531-psrvh\" (UID: \"0f84c2a9-a227-45bc-8690-b610658bff5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:07.733374 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.733347    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"]
Nov 02 14:01:07.744703 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.744676    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:01:07.744874 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.744765    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 14:01:07.826761 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.826719    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dafa3531-psrvh\" (UID: \"0f84c2a9-a227-45bc-8690-b610658bff5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:07.827007 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.826777    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj5pk\" (UniqueName: \"kubernetes.io/projected/0f84c2a9-a227-45bc-8690-b610658bff5f-kube-api-access-cj5pk\") pod \"maas-api-revoke-sub-dafa3531-psrvh\" (UID: \"0f84c2a9-a227-45bc-8690-b610658bff5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:07.827007 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.826828    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dafa3531-psrvh\" (UID: \"0f84c2a9-a227-45bc-8690-b610658bff5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:07.827433 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.827407    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dafa3531-psrvh\" (UID: \"0f84c2a9-a227-45bc-8690-b610658bff5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:07.829462 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.829436    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dafa3531-psrvh\" (UID: \"0f84c2a9-a227-45bc-8690-b610658bff5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:07.852615 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:07.852577    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj5pk\" (UniqueName: \"kubernetes.io/projected/0f84c2a9-a227-45bc-8690-b610658bff5f-kube-api-access-cj5pk\") pod \"maas-api-revoke-sub-dafa3531-psrvh\" (UID: \"0f84c2a9-a227-45bc-8690-b610658bff5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:08.026767 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:08.026676    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:08.330877 ip-10-0-134-229 kubenswrapper[5879]: W1102 14:01:08.330846    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f84c2a9_a227_45bc_8690_b610658bff5f.slice/crio-cec8d211112b5b1dac41f07dc5eb231dee5d62db04a94af624e9146f904c6d56 WatchSource:0}: Error finding container cec8d211112b5b1dac41f07dc5eb231dee5d62db04a94af624e9146f904c6d56: Status 404 returned error can't find the container with id cec8d211112b5b1dac41f07dc5eb231dee5d62db04a94af624e9146f904c6d56
Nov 02 14:01:08.332806 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:08.332696    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"]
Nov 02 14:01:09.092632 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:09.092592    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh" event={"ID":"0f84c2a9-a227-45bc-8690-b610658bff5f","Type":"ContainerStarted","Data":"3668cec2dd5a0b2dbacba3ac066e16e8f5302793272cb0fa9fc43bca89689c34"}
Nov 02 14:01:09.093193 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:09.092653    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh" event={"ID":"0f84c2a9-a227-45bc-8690-b610658bff5f","Type":"ContainerStarted","Data":"cec8d211112b5b1dac41f07dc5eb231dee5d62db04a94af624e9146f904c6d56"}
Nov 02 14:01:09.138730 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:09.138664    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh" podStartSLOduration=2.138650187 podStartE2EDuration="2.138650187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:09.137425768 +0000 UTC m=+5615.244128616" watchObservedRunningTime="2026-11-02 14:01:09.138650187 +0000 UTC m=+5615.245353037"
Nov 02 14:01:10.127781 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:10.127743    5879 generic.go:356] "Generic (PLEG): container finished" podID="0f84c2a9-a227-45bc-8690-b610658bff5f" containerID="3668cec2dd5a0b2dbacba3ac066e16e8f5302793272cb0fa9fc43bca89689c34" exitCode=0
Nov 02 14:01:10.139365 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:10.128065    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh" event={"ID":"0f84c2a9-a227-45bc-8690-b610658bff5f","Type":"ContainerDied","Data":"3668cec2dd5a0b2dbacba3ac066e16e8f5302793272cb0fa9fc43bca89689c34"}
Nov 02 14:01:11.041019 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:11.040186    5879 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-9tbj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 14:01:11.041019 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:11.040261    5879 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-9tbj6" podUID="3a8de801-f2f2-4b08-ad24-022a2e063bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 14:01:11.474095 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:11.474067    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:11.569065 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:11.569023    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-openshift-ca\") pod \"0f84c2a9-a227-45bc-8690-b610658bff5f\" (UID: \"0f84c2a9-a227-45bc-8690-b610658bff5f\") "
Nov 02 14:01:11.569360 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:11.569147    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-xks-ca\") pod \"0f84c2a9-a227-45bc-8690-b610658bff5f\" (UID: \"0f84c2a9-a227-45bc-8690-b610658bff5f\") "
Nov 02 14:01:11.569360 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:11.569231    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f84c2a9-a227-45bc-8690-b610658bff5f-kube-api-access-cj5pk\" (UniqueName: \"kubernetes.io/projected/0f84c2a9-a227-45bc-8690-b610658bff5f-kube-api-access-cj5pk\") pod \"0f84c2a9-a227-45bc-8690-b610658bff5f\" (UID: \"0f84c2a9-a227-45bc-8690-b610658bff5f\") "
Nov 02 14:01:11.569587 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:11.569550    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-openshift-ca" pod "0f84c2a9-a227-45bc-8690-b610658bff5f" (UID: "0f84c2a9-a227-45bc-8690-b610658bff5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:11.572225 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:11.572198    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-xks-ca" pod "0f84c2a9-a227-45bc-8690-b610658bff5f" (UID: "0f84c2a9-a227-45bc-8690-b610658bff5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:01:11.572519 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:11.572496    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f84c2a9-a227-45bc-8690-b610658bff5f-kube-api-access-cj5pk" pod "0f84c2a9-a227-45bc-8690-b610658bff5f" (UID: "0f84c2a9-a227-45bc-8690-b610658bff5f"). InnerVolumeSpecName "kube-api-access-cj5pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:11.670099 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:11.670046    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:01:11.670099 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:11.670098    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f84c2a9-a227-45bc-8690-b610658bff5f-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:01:11.670099 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:11.670111    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj5pk\" (UniqueName: \"kubernetes.io/projected/0f84c2a9-a227-45bc-8690-b610658bff5f-kube-api-access-cj5pk\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:01:12.165779 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:12.165744    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh"
Nov 02 14:01:12.165779 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:12.165768    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafa3531-psrvh" event={"ID":"0f84c2a9-a227-45bc-8690-b610658bff5f","Type":"ContainerDied","Data":"cec8d211112b5b1dac41f07dc5eb231dee5d62db04a94af624e9146f904c6d56"}
Nov 02 14:01:12.166043 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:12.165818    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cec8d211112b5b1dac41f07dc5eb231dee5d62db04a94af624e9146f904c6d56"
Nov 02 14:01:14.821692 ip-10-0-134-229 kubenswrapper[5879]: E1102 14:01:14.821651    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 14:01:18.059628 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.059591    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"]
Nov 02 14:01:18.060103 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.060043    5879 cpu_manager.go:414] "containerMap: removing container" podUID="0f84c2a9-a227-45bc-8690-b610658bff5f" containerName="revoke-keys"
Nov 02 14:01:18.060103 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.060056    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f84c2a9-a227-45bc-8690-b610658bff5f" containerName="revoke-keys"
Nov 02 14:01:18.060200 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.060124    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f84c2a9-a227-45bc-8690-b610658bff5f" containerName="revoke-keys"
Nov 02 14:01:18.063489 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.063468    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:18.087479 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.087450    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"]
Nov 02 14:01:18.097937 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.097911    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 14:01:18.097937 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.097912    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-45fzn\""
Nov 02 14:01:18.135709 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.135668    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e948b155-7n9p9\" (UID: \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:18.135890 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.135725    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmvbg\" (UniqueName: \"kubernetes.io/projected/ee77fd16-3eb8-488a-b762-a2a15d0752fd-kube-api-access-zmvbg\") pod \"maas-api-revoke-sub-e948b155-7n9p9\" (UID: \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:18.135946 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.135884    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e948b155-7n9p9\" (UID: \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:18.237399 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.237365    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e948b155-7n9p9\" (UID: \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:18.237621 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.237421    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmvbg\" (UniqueName: \"kubernetes.io/projected/ee77fd16-3eb8-488a-b762-a2a15d0752fd-kube-api-access-zmvbg\") pod \"maas-api-revoke-sub-e948b155-7n9p9\" (UID: \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:18.237621 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.237503    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e948b155-7n9p9\" (UID: \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:18.238265 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.238239    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e948b155-7n9p9\" (UID: \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:18.240107 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.240082    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e948b155-7n9p9\" (UID: \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:18.263225 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.263196    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmvbg\" (UniqueName: \"kubernetes.io/projected/ee77fd16-3eb8-488a-b762-a2a15d0752fd-kube-api-access-zmvbg\") pod \"maas-api-revoke-sub-e948b155-7n9p9\" (UID: \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:18.376126 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.376099    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:18.677932 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:18.677896    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"]
Nov 02 14:01:18.678090 ip-10-0-134-229 kubenswrapper[5879]: W1102 14:01:18.677946    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee77fd16_3eb8_488a_b762_a2a15d0752fd.slice/crio-5012a9789775f31470d341eb369cff4f05a8a07b49cde5341d4058508ff41b75 WatchSource:0}: Error finding container 5012a9789775f31470d341eb369cff4f05a8a07b49cde5341d4058508ff41b75: Status 404 returned error can't find the container with id 5012a9789775f31470d341eb369cff4f05a8a07b49cde5341d4058508ff41b75
Nov 02 14:01:19.207389 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:19.207348    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9" event={"ID":"ee77fd16-3eb8-488a-b762-a2a15d0752fd","Type":"ContainerStarted","Data":"ebf51d99534b97292ed345dfcadf99010e034a24caf98284ae86944d8bfb5d14"}
Nov 02 14:01:19.207804 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:19.207414    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9" event={"ID":"ee77fd16-3eb8-488a-b762-a2a15d0752fd","Type":"ContainerStarted","Data":"5012a9789775f31470d341eb369cff4f05a8a07b49cde5341d4058508ff41b75"}
Nov 02 14:01:19.252770 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:19.252669    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9" podStartSLOduration=1.2526565619999999 podStartE2EDuration="1.252656562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:01:19.251504459 +0000 UTC m=+5625.358207310" watchObservedRunningTime="2026-11-02 14:01:19.252656562 +0000 UTC m=+5625.359359413"
Nov 02 14:01:20.236019 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:20.235948    5879 generic.go:356] "Generic (PLEG): container finished" podID="ee77fd16-3eb8-488a-b762-a2a15d0752fd" containerID="ebf51d99534b97292ed345dfcadf99010e034a24caf98284ae86944d8bfb5d14" exitCode=0
Nov 02 14:01:20.236654 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:20.236206    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9" event={"ID":"ee77fd16-3eb8-488a-b762-a2a15d0752fd","Type":"ContainerDied","Data":"ebf51d99534b97292ed345dfcadf99010e034a24caf98284ae86944d8bfb5d14"}
Nov 02 14:01:20.655133 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:20.655095    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 14:01:20.778283 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:20.778153    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 14:01:21.392735 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:21.392708    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:21.478152 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:21.478114    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-openshift-ca\") pod \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\" (UID: \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\") "
Nov 02 14:01:21.478152 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:21.478155    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-xks-ca\") pod \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\" (UID: \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\") "
Nov 02 14:01:21.478374 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:21.478257    5879 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee77fd16-3eb8-488a-b762-a2a15d0752fd-kube-api-access-zmvbg\" (UniqueName: \"kubernetes.io/projected/ee77fd16-3eb8-488a-b762-a2a15d0752fd-kube-api-access-zmvbg\") pod \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\" (UID: \"ee77fd16-3eb8-488a-b762-a2a15d0752fd\") "
Nov 02 14:01:21.478519 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:21.478496    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-openshift-ca" pod "ee77fd16-3eb8-488a-b762-a2a15d0752fd" (UID: "ee77fd16-3eb8-488a-b762-a2a15d0752fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 14:01:21.478618 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:21.478604    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-openshift-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:01:21.480452 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:21.480429    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-xks-ca" pod "ee77fd16-3eb8-488a-b762-a2a15d0752fd" (UID: "ee77fd16-3eb8-488a-b762-a2a15d0752fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 14:01:21.480550 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:21.480526    5879 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee77fd16-3eb8-488a-b762-a2a15d0752fd-kube-api-access-zmvbg" pod "ee77fd16-3eb8-488a-b762-a2a15d0752fd" (UID: "ee77fd16-3eb8-488a-b762-a2a15d0752fd"). InnerVolumeSpecName "kube-api-access-zmvbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 14:01:21.579750 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:21.579646    5879 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee77fd16-3eb8-488a-b762-a2a15d0752fd-maas-api-xks-ca\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:01:21.579750 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:21.579695    5879 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmvbg\" (UniqueName: \"kubernetes.io/projected/ee77fd16-3eb8-488a-b762-a2a15d0752fd-kube-api-access-zmvbg\") on node \"ip-10-0-134-229.ec2.internal\" DevicePath \"\""
Nov 02 14:01:22.250288 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:22.250254    5879 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9"
Nov 02 14:01:22.250288 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:22.250273    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e948b155-7n9p9" event={"ID":"ee77fd16-3eb8-488a-b762-a2a15d0752fd","Type":"ContainerDied","Data":"5012a9789775f31470d341eb369cff4f05a8a07b49cde5341d4058508ff41b75"}
Nov 02 14:01:22.250510 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:22.250319    5879 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5012a9789775f31470d341eb369cff4f05a8a07b49cde5341d4058508ff41b75"
Nov 02 14:01:28.256830 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:28.256788    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 14:01:43.808320 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:43.808278    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 14:01:44.820376 ip-10-0-134-229 kubenswrapper[5879]: E1102 14:01:44.820342    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 14:01:45.490975 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:45.490941    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 14:01:50.863550 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:50.863520    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-7js7d_2f09ae13-0192-4a68-a935-39846f02caca/manager/0.log"
Nov 02 14:01:51.059660 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:51.059626    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-klw92_0b2da242-fe89-4999-a164-2f251fb97b3f/manager/0.log"
Nov 02 14:01:51.269087 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:51.268982    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vd4d6_8e9e3acc-7a16-4166-bf7d-22cc70517482/manager/0.log"
Nov 02 14:01:51.772793 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:51.772756    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nwfcr_8891b09c-a283-422f-a4ea-25f84ea846a8/server/0.log"
Nov 02 14:01:51.958084 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:51.958053    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-vwd9x_8b097954-45fc-45e8-937e-4ece9fbdc1c6/manager/0.log"
Nov 02 14:01:52.175463 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:52.175435    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cvvr4_637730bd-76f5-4bff-8dc2-ba86e4e5254d/proxy/0.log"
Nov 02 14:01:53.639656 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:53.639620    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2135802a-chnnp_f22c82c7-938a-47d9-bf58-6d204d63794f/revoke-keys/0.log"
Nov 02 14:01:53.834067 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:53.834031    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4fe77a7a-p2lsr_e28783d4-5446-4802-a9e1-134b867cd015/revoke-keys/0.log"
Nov 02 14:01:54.017283 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:54.017193    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5c3e49de-wht8g_b696b31d-74b1-4e13-a1e7-70d947a62116/revoke-keys/0.log"
Nov 02 14:01:54.200371 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:54.200340    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5efb4fe6-jm2zd_fb11133c-62d7-4b85-acbb-0c3f7dbdb1d9/revoke-keys/0.log"
Nov 02 14:01:54.376715 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:54.376692    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6005186d-mjkxs_222e7c04-54d7-4429-9bf2-3943f5e03922/revoke-keys/0.log"
Nov 02 14:01:54.774728 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:54.774652    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-85d5088e-5ljj5_399ee53f-cec5-4d50-80b7-afaa8e3aff94/revoke-keys/0.log"
Nov 02 14:01:54.975774 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:54.975742    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-93219489-287gv_dffdf00f-b452-499b-b760-ddeedf3d3aad/revoke-keys/0.log"
Nov 02 14:01:55.164180 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:55.164143    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a0692b79-56tjj_1240f278-eefe-4caa-92a5-e07818114e29/revoke-keys/0.log"
Nov 02 14:01:55.343527 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:55.343493    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bf8f1462-tg7gn_535905be-87fb-4071-a92d-416703c0a626/revoke-keys/0.log"
Nov 02 14:01:55.537369 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:55.537277    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c5cf680c-psslk_0dda6167-10a1-4679-821b-fd09ca5eccf5/revoke-keys/0.log"
Nov 02 14:01:55.726738 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:55.726708    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d60ead89-xtgqz_460dd41d-abbd-43ee-a179-066dd1920fbb/revoke-keys/0.log"
Nov 02 14:01:55.914083 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:55.914053    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-daf436c7-cc899_ecd90196-10cd-4640-bd9c-318dac558ef5/revoke-keys/0.log"
Nov 02 14:01:56.109554 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:56.109522    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dafa3531-psrvh_0f84c2a9-a227-45bc-8690-b610658bff5f/revoke-keys/0.log"
Nov 02 14:01:56.535792 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:56.535761    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e948b155-7n9p9_ee77fd16-3eb8-488a-b762-a2a15d0752fd/revoke-keys/0.log"
Nov 02 14:01:56.730391 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:56.730356    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-eda01314-bsvwv_e0baebe0-7263-4aec-9fbd-96b0a5d5fa2c/revoke-keys/0.log"
Nov 02 14:01:56.938100 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:56.938064    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f6bc25c9-jspdr_725153ba-0bf2-4c41-bc02-6f5db080c686/revoke-keys/0.log"
Nov 02 14:01:57.124754 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:57.124723    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fefb5b75-j5c6v_f3f04f59-c343-4387-944a-6caf51ef51e2/revoke-keys/0.log"
Nov 02 14:01:57.312047 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:57.311948    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-kzdf5_aae37103-208d-4ba4-a142-f5a29c9fdc37/postgres/0.log"
Nov 02 14:01:59.059017 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:59.058938    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"]
Nov 02 14:01:59.068864 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:59.068827    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d60ead89-xtgqz"]
Nov 02 14:01:59.851472 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:01:59.851438    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rdc2r_d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb/kuadrant-console-plugin/0.log"
Nov 02 14:02:00.015203 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:00.015157    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="460dd41d-abbd-43ee-a179-066dd1920fbb" path="/var/lib/kubelet/pods/460dd41d-abbd-43ee-a179-066dd1920fbb/volumes"
Nov 02 14:02:02.110553 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:02.110516    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c9fcfcf5-hjsfn_def76310-095a-4ed5-a40d-983f53799e40/kube-auth-proxy/0.log"
Nov 02 14:02:02.304244 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:02.304210    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-27twq_c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d/istio-proxy/0.log"
Nov 02 14:02:02.911161 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:02.911127    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-9tbj6_3a8de801-f2f2-4b08-ad24-022a2e063bc8/istio-proxy/0.log"
Nov 02 14:02:03.940260 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:03.940229    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-m2dn7_e1ec3a45-ae38-4297-a894-27438a341dcf/istio-proxy/0.log"
Nov 02 14:02:09.041472 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:09.041437    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"]
Nov 02 14:02:09.054736 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:09.054695    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0692b79-56tjj"]
Nov 02 14:02:10.015695 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:10.015664    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1240f278-eefe-4caa-92a5-e07818114e29" path="/var/lib/kubelet/pods/1240f278-eefe-4caa-92a5-e07818114e29/volumes"
Nov 02 14:02:14.820839 ip-10-0-134-229 kubenswrapper[5879]: E1102 14:02:14.820754    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 14:02:17.810473 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:17.810435    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:17.988138 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:17.988102    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:22.799612 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:22.799579    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x7jpb_259d31ef-0ef7-4655-9121-88fd07468c38/global-pull-secret-syncer/0.log"
Nov 02 14:02:23.484464 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:23.484429    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log"
Nov 02 14:02:24.120644 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:24.120604    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-229.ec2.internal_f2aebf6f2caaa3c2095ff6e33664045c/haproxy/0.log"
Nov 02 14:02:31.046384 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:31.046340    5879 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"]
Nov 02 14:02:31.060153 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:31.060115    5879 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2135802a-chnnp"]
Nov 02 14:02:31.203595 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:31.203566    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-rdc2r_d6668a9b-ef69-4bfc-a4a4-5ff66c0b62bb/kuadrant-console-plugin/0.log"
Nov 02 14:02:32.015845 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.015809    5879 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22c82c7-938a-47d9-bf58-6d204d63794f" path="/var/lib/kubelet/pods/f22c82c7-938a-47d9-bf58-6d204d63794f/volumes"
Nov 02 14:02:32.327219 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.327109    5879 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"]
Nov 02 14:02:32.327699 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.327684    5879 cpu_manager.go:414] "containerMap: removing container" podUID="ee77fd16-3eb8-488a-b762-a2a15d0752fd" containerName="revoke-keys"
Nov 02 14:02:32.327746 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.327702    5879 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee77fd16-3eb8-488a-b762-a2a15d0752fd" containerName="revoke-keys"
Nov 02 14:02:32.327817 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.327805    5879 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee77fd16-3eb8-488a-b762-a2a15d0752fd" containerName="revoke-keys"
Nov 02 14:02:32.331269 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.331252    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.358114 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.358083    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"]
Nov 02 14:02:32.369510 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.369480    5879 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-trmxf\"/\"default-dockercfg-p5wzj\""
Nov 02 14:02:32.425509 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.425473    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-sys\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.425509 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.425514    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pq6l\" (UniqueName: \"kubernetes.io/projected/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-kube-api-access-2pq6l\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.425769 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.425631    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-lib-modules\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.425769 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.425662    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-podres\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.425769 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.425685    5879 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-proc\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.432974 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.432941    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trmxf\"/\"kube-root-ca.crt\""
Nov 02 14:02:32.432974 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.432941    5879 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-trmxf\"/\"openshift-service-ca.crt\""
Nov 02 14:02:32.526513 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.526468    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-lib-modules\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.526513 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.526511    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-podres\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.526743 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.526547    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-proc\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.526743 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.526625    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-lib-modules\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.526743 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.526629    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-proc\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.526743 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.526629    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-podres\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.526743 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.526725    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-sys\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.526926 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.526754    5879 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pq6l\" (UniqueName: \"kubernetes.io/projected/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-kube-api-access-2pq6l\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.526926 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.526833    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-sys\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.552771 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.552731    5879 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pq6l\" (UniqueName: \"kubernetes.io/projected/8802fcbd-591b-4bda-8d40-e0fc99d7bbff-kube-api-access-2pq6l\") pod \"perf-node-gather-daemonset-dk68w\" (UID: \"8802fcbd-591b-4bda-8d40-e0fc99d7bbff\") " pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.645096 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.645056    5879 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:32.948276 ip-10-0-134-229 kubenswrapper[5879]: W1102 14:02:32.948241    5879 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8802fcbd_591b_4bda_8d40_e0fc99d7bbff.slice/crio-634a578076c2a9c7a5760b981507059c9b713df0c6a697374bee2ffe843dc762 WatchSource:0}: Error finding container 634a578076c2a9c7a5760b981507059c9b713df0c6a697374bee2ffe843dc762: Status 404 returned error can't find the container with id 634a578076c2a9c7a5760b981507059c9b713df0c6a697374bee2ffe843dc762
Nov 02 14:02:32.948852 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.948825    5879 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"]
Nov 02 14:02:32.994539 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:32.994515    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff446abc-cc59-403b-bc19-3b1d514b19a3/alertmanager/0.log"
Nov 02 14:02:33.124549 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:33.124518    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff446abc-cc59-403b-bc19-3b1d514b19a3/config-reloader/0.log"
Nov 02 14:02:33.247560 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:33.247467    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff446abc-cc59-403b-bc19-3b1d514b19a3/kube-rbac-proxy-web/0.log"
Nov 02 14:02:33.372405 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:33.372373    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff446abc-cc59-403b-bc19-3b1d514b19a3/kube-rbac-proxy/0.log"
Nov 02 14:02:33.497466 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:33.497434    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff446abc-cc59-403b-bc19-3b1d514b19a3/kube-rbac-proxy-metric/0.log"
Nov 02 14:02:33.628796 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:33.628761    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff446abc-cc59-403b-bc19-3b1d514b19a3/prom-label-proxy/0.log"
Nov 02 14:02:33.686337 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:33.686303    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w" event={"ID":"8802fcbd-591b-4bda-8d40-e0fc99d7bbff","Type":"ContainerStarted","Data":"166e15fd3b2256ad4446db73d2d67cfa2c04ff008c20b78dfc25a45644fd8aad"}
Nov 02 14:02:33.686517 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:33.686353    5879 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w" event={"ID":"8802fcbd-591b-4bda-8d40-e0fc99d7bbff","Type":"ContainerStarted","Data":"634a578076c2a9c7a5760b981507059c9b713df0c6a697374bee2ffe843dc762"}
Nov 02 14:02:33.686564 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:33.686515    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:33.731543 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:33.731493    5879 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w" podStartSLOduration=1.731477828 podStartE2EDuration="1.731477828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:02:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:02:33.730560932 +0000 UTC m=+5699.837263784" watchObservedRunningTime="2026-11-02 14:02:33.731477828 +0000 UTC m=+5699.838180679"
Nov 02 14:02:33.749006 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:33.748972    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff446abc-cc59-403b-bc19-3b1d514b19a3/init-config-reloader/0.log"
Nov 02 14:02:34.001573 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:34.001473    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-v57wr_ff9623fb-ce84-42fc-aa52-fc263d66ec90/kube-state-metrics/0.log"
Nov 02 14:02:34.117314 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:34.117275    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-v57wr_ff9623fb-ce84-42fc-aa52-fc263d66ec90/kube-rbac-proxy-main/0.log"
Nov 02 14:02:34.239021 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:34.238961    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-v57wr_ff9623fb-ce84-42fc-aa52-fc263d66ec90/kube-rbac-proxy-self/0.log"
Nov 02 14:02:34.363418 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:34.363337    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f7db9d655-n8z5r_14492824-39e2-4e98-a91d-ac81382ad70f/metrics-server/0.log"
Nov 02 14:02:34.487833 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:34.487802    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-5d9mm_c0067039-d47d-4ee4-8e02-1e24cdc26732/monitoring-plugin/0.log"
Nov 02 14:02:35.006493 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:35.006460    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lbm4n_d9215ec4-711e-400c-852c-aaaebbd7a7ec/node-exporter/0.log"
Nov 02 14:02:35.126678 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:35.126645    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lbm4n_d9215ec4-711e-400c-852c-aaaebbd7a7ec/kube-rbac-proxy/0.log"
Nov 02 14:02:35.246085 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:35.246054    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lbm4n_d9215ec4-711e-400c-852c-aaaebbd7a7ec/init-textfile/0.log"
Nov 02 14:02:35.760224 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:35.760193    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-x9flx_5bb243e5-8e99-4391-9a55-6e3a8f82ae0c/kube-rbac-proxy-main/0.log"
Nov 02 14:02:35.881344 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:35.881308    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-x9flx_5bb243e5-8e99-4391-9a55-6e3a8f82ae0c/kube-rbac-proxy-self/0.log"
Nov 02 14:02:36.010499 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:36.010410    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-x9flx_5bb243e5-8e99-4391-9a55-6e3a8f82ae0c/openshift-state-metrics/0.log"
Nov 02 14:02:36.145802 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:36.145774    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b834743c-3e58-4455-bbad-5f1a0ee4ae50/prometheus/0.log"
Nov 02 14:02:36.268194 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:36.268111    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b834743c-3e58-4455-bbad-5f1a0ee4ae50/config-reloader/0.log"
Nov 02 14:02:36.389121 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:36.389093    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b834743c-3e58-4455-bbad-5f1a0ee4ae50/thanos-sidecar/0.log"
Nov 02 14:02:36.510110 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:36.510076    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b834743c-3e58-4455-bbad-5f1a0ee4ae50/kube-rbac-proxy-web/0.log"
Nov 02 14:02:36.637230 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:36.637204    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b834743c-3e58-4455-bbad-5f1a0ee4ae50/kube-rbac-proxy/0.log"
Nov 02 14:02:36.758232 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:36.758208    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b834743c-3e58-4455-bbad-5f1a0ee4ae50/kube-rbac-proxy-thanos/0.log"
Nov 02 14:02:36.883938 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:36.883907    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b834743c-3e58-4455-bbad-5f1a0ee4ae50/init-config-reloader/0.log"
Nov 02 14:02:37.390222 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:37.390190    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cf7f64998-5wcsl_025ece4b-9534-475f-b96d-0cb3edfe3ac3/telemeter-client/0.log"
Nov 02 14:02:37.510944 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:37.510911    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cf7f64998-5wcsl_025ece4b-9534-475f-b96d-0cb3edfe3ac3/reload/0.log"
Nov 02 14:02:37.633567 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:37.633530    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-cf7f64998-5wcsl_025ece4b-9534-475f-b96d-0cb3edfe3ac3/kube-rbac-proxy/0.log"
Nov 02 14:02:37.773224 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:37.773127    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f985446f-hkswt_3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/thanos-query/0.log"
Nov 02 14:02:37.892453 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:37.892416    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f985446f-hkswt_3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/kube-rbac-proxy-web/0.log"
Nov 02 14:02:38.012596 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:38.012562    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f985446f-hkswt_3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/kube-rbac-proxy/0.log"
Nov 02 14:02:38.134203 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:38.134179    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f985446f-hkswt_3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/prom-label-proxy/0.log"
Nov 02 14:02:38.256465 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:38.256434    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f985446f-hkswt_3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/kube-rbac-proxy-rules/0.log"
Nov 02 14:02:38.379814 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:38.379781    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57f985446f-hkswt_3cc7e7ac-ba25-45f4-b6c9-5d7bff402d72/kube-rbac-proxy-metrics/0.log"
Nov 02 14:02:39.702887 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:39.702856    5879 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-trmxf/perf-node-gather-daemonset-dk68w"
Nov 02 14:02:44.820783 ip-10-0-134-229 kubenswrapper[5879]: E1102 14:02:44.820736    5879 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8mqtc" containerName="extractor"
Nov 02 14:02:45.673961 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:45.673910    5879 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-26p8v_83dcdc7a-f1ba-4570-bed5-a40b0d3b213d/konnectivity-agent/0.log" limit=1024
Nov 02 14:02:46.103856 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:46.103762    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-snnjx_9b72267e-e7f7-4115-af3e-14a382e659af/dns/0.log"
Nov 02 14:02:46.229164 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:46.229134    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-snnjx_9b72267e-e7f7-4115-af3e-14a382e659af/kube-rbac-proxy/0.log"
Nov 02 14:02:46.638942 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:46.638909    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k49jr_b89c2be7-3500-4d9a-ac82-4ef367b83e3e/dns-node-resolver/0.log"
Nov 02 14:02:48.002268 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:48.002235    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9c9fcfcf5-hjsfn_def76310-095a-4ed5-a40d-983f53799e40/kube-auth-proxy/0.log"
Nov 02 14:02:48.078137 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:48.078102    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-27twq_c01e2f4e-f5ca-4231-bc14-be2fa6b4b59d/istio-proxy/0.log"
Nov 02 14:02:48.194423 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:48.194395    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mdzbn_d59c905e-0042-4777-a2fe-d5d5ae04e47d/node-ca/0.log"
Nov 02 14:02:48.290512 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:48.290422    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-9tbj6_3a8de801-f2f2-4b08-ad24-022a2e063bc8/istio-proxy/0.log"
Nov 02 14:02:48.637502 ip-10-0-134-229 kubenswrapper[5879]: I1102 14:02:48.637459    5879 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-m2dn7_e1ec3a45-ae38-4297-a894-27438a341dcf/istio-proxy/0.log"